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Abstract 


This  work  represents  the  fourth  phase  of  a  project  investigating  the  Canadian  Forces  (CF) 
Operational  Planning  Process  (OPP)  and  an  alternative  planning  process  based  on  intuitive 
decision  making.  This  is  in  support  of  a  larger  project,  Project  Minerva,  focused  on  re¬ 
examining  Command  and  Control  (C2),  specifically  the  CF  OPP,  in  the  Land  Force  in  light  of 
the  implementation  of  digitized  C2  systems.  The  CF  OPP  represents  an  analytic  decision  making 
process  in  which  1)  multiple  solutions  to  the  problem  must  be  evaluated  and  the  best  selected, 
and  2)  evaluation  of  solution  alternatives  must  be  performed  through  exhaustive  factor-by-factor 
comparison.  Research  in  the  cognitive  sciences  has  suggested  that  a  large  portion  of  human 
decision  making  is  conducted  intuitively;  i.e.  by  less  formal,  non-analytic  processes.  Thus,  there 
may  be  a  mismatch  between  the  OPP  as  laid  out  in  doctrine  and  taught  at  training  and  education 
institutions  within  the  CF,  and  the  planning  process  as  practiced  by  command  teams  in  more 
operational  settings,  especially  at  the  Brigade  level  and  below. 

Specifically,  the  current  work  includes  the  development  of  an  alternative  planning  process  based 
on  intuitive  decision  making  (referred  to  as  the  Intuitive  Operations  Planning  Process  or  IOPP), 
the  development  of  a  training  course  for  the  IOPP,  and  an  evaluation  of  the  effectiveness  of  the 
IOPP  compared  to  the  existing  CF  OPP. 

The  IOPP  exhibits  the  best  characteristics  of  other  intuitive  planning  models  (Kievenaar,  1997; 
Schmitt  &  Klein,  1999;  Thunholm,  2005;  Whitehurst,  2002)  and  incorporates  findings  from 
previous  work  investigating  application  of  the  OPP  in  the  CF  (Bruyn  et  al.,  2005),  while 
maintaining  a  large  amount  of  the  terminology,  outputs  generated  and  formal  staff  briefings  used 
in  the  OPP  in  order  to  promote  level  of  acceptance  by  CF  practitioners  and  face  validity  of  the 
IOPP. 

A  web-based  training  course  for  both  the  OPP  and  IOPP  was  developed  as  it  allows  more 
flexibility  in  terms  of  the  delivery  of  the  training  than  lecture-based  training.  Aside  from  some 
minor  technological  complications,  delivery  of  the  training  courses  proved  to  be  successful. 
Participants  provided  valuable  feedback  in  terms  of  training  content  and  delivery  that  should  be 
taken  into  account  in  future  training  enhancements 

The  first  iteration  of  the  IOPP  model  resulted  in  positive  feedback  from  participants  in  the 
experimental  evaluation.  It  was  also  found  that  the  IOPP  outperformed  the  OPP  in  terms  of 
efficiency,  number  of  steps  performed,  quality  of  planning  products,  usability  of  planning 
process,  operational  effectiveness,  and  participant  workload  levels.  On  the  other  hand, 
participants’  perceived  level  of  trust  and  reliability  was  lower  for  the  IOPP  compared  to  the  OPP. 
They  also  felt  that  the  IOPP  was  more  likely  to  lead  to  errors  (and  more  critical  errors)  in  a  Plan. 
Participants  also  provided  a  number  of  benefits  and  limitations  of  the  IOPP,  which  should  be 
considered  in  future  research. 
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Resume 


Ce  travail  est  la  quatrieme  phase  d’un  projet  d’enquete  sur  le  processus  de  planification 
operationnelle  (PPO)  des  Forces  canadiennes  (FC)  et  un  processus  de  planification  de  rechange 
fonde  sur  la  prise  de  decisions  intuitives.  II  vise  a  appuyer  un  projet  plus  vaste,  le  projet 
Minerve,  qui  est  axe  sur  un  nouvel  examen  du  commandement  et  controle  (C2)  en  particulier  le 
PPO  FC  dans  la  force  terrestre,  compte  tenu  de  la  mise  en  place  des  systemes  C2  numerises.  Le 
PPO  FC  represente  un  processus  de  prise  de  decision  analytique  au  cours  duquel  1)  plusieurs 
solutions  au  probleme  doivent  etre  evaluees  et  la  meilleure  solution  choisie  et  2)  revaluation  de 
solutions  de  rechange  doit  etre  effectuee  par  une  comparaison  exhaustive  critere  par  critere.  Les 
recherches  en  sciences  cognitives  indiquent  qu’une  grande  partie  de  la  prise  de  decision  humaine 
est  intuitive,  c’est-a-dire  qu’elle  s’effectue  selon  un  processus  moins  analytique,  moins  formel. 
Par  consequent,  il  peut  y  avoir  une  discordance  entre  le  PPO  tel  que  decrit  dans  la  doctrine  et 
enseigne  dans  les  etablissements  d’education  et  d’instruction  des  FC  et  le  processus  de 
planification  tel  que  mis  en  pratique  par  l’equipe  de  commandement  dans  les  contextes  plus 
operationnels,  en  particulier  a  la  brigade  et  aux  niveaux  inferieurs. 

Plus  particulierement,  ce  travail  comprend  l’elaboration  d’un  processus  de  planification  de 
rechange  fonde  sur  la  prise  de  decisions  intuitives  (appele  Processus  de  planification  intuitive  des 
operations  ou  PPIO),  la  preparation  d’un  cours  sur  le  PPIO  et  une  evaluation  de  l’efficacite  du 
PPIO  par  rapport  au  PPO  FC  actuel. 

Le  PPIO  presente  les  meilleures  caracteristiques  d’autres  modeles  de  planification  intuitive 
(Kievenaar,  1997;  Schmitt  &  Klein,  1999;  Thunholm,  2005;  Whitehurst,  2002)  et  inclut  des 
resultats  de  travaux  de  recherche  precedents  portant  sur  la  mise  en  oeuvre  du  PPO  dans  les  FC 
(Bruyn  et  coll.,  2005),  tout  en  conservant  une  grande  partie  de  la  terminologie  du  PPO,  les 
commentaires  generes  par  le  PPO  et  des  seances  d’information  officielles  employees  dans  le 
cadre  de  ce  processus  afin  de  favoriser  le  degre  d’ acceptation  des  utilisateurs  des  FC  et  la  validite 
apparente  du  PPIO. 

Un  cours  en  ligne  sur  le  PPO  et  le  PPIO  a  ete  prepare  car  il  permet  une  plus  grande  souplesse 
concernant  la  prestation  de  l’instruction  qu’un  cours  en  salle  de  classe.  Outre  quelques 
complications  technologiques,  la  prestation  de  l’instruction  s’est  averee  un  succes.  Les 
participants  ont  fait  des  commentaires  utiles  sur  le  contenu  de  l’instruction  et  la  prestation  dont  il 
faudrait  tenir  compte  lors  des  prochaines  revisions  de  1’ instruction 

La  premiere  iteration  du  modele  PPIO  a  suscite  des  commentaires  positifs  de  la  part  des 
participants  au  cours  de  1’evaluation  experimental .  On  a  aussi  constate  que  le  PPIO  est  superieur 
au  PPO  en  ce  qui  a  trait  a  l’efficacite,  au  nombre  d’etapes  necessaires,  a  la  qualite  des  produits 
de  planification,  a  la  convivialite  du  processus  de  planification,  a  l’efficacite  operationnelle  et  a  la 
charge  de  travail  des  participants.  D’autre  part,  les  participants  ont  trouve  que  le  niveau  de 
confiance  et  de  fiabilite  du  PPIO  etait  inferieur  a  celui  du  PPO.  Ils  ont  aussi  eu  l’impression  que 
le  PPIO  risquait  davantage  de  dormer  lieu  a  des  erreurs  (et  des  erreurs  plus  critiques)  dans  un 
plan.  Les  participants  ont  aussi  donne  les  avantages  et  les  limites  du  PPIO  dont  il  faudra  tenir 
compte  lors  des  prochaines  recherches. 
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Executive  Summary 

This  work  represents  the  fourth  phase  of  a  project  investigating  the  Canadian  Forces  (CF) 
Operational  Planning  Process  (OPP)  and  an  alternative  planning  process  based  on  intuitive 
decision  making.  This  is  in  support  of  a  larger  project,  Project  Minerva,  focused  on  re¬ 
examining  Command  and  Control  (C2),  specifically  the  CF  OPP,  in  the  Land  Force  in  light  of 
the  implementation  of  digitized  C2  systems.  The  CF  OPP  represents  an  analytic  decision  making 
process  in  which  1)  multiple  solutions  to  the  problem  must  be  evaluated  and  the  best  selected, 
and  2)  evaluation  of  solution  alternatives  must  be  performed  through  exhaustive  factor-by-factor 
comparison.  Research  in  the  cognitive  sciences  has  suggested  that  a  large  portion  of  human 
decision  making  is  conducted  intuitively;  i.e.  by  less  formal,  non-analytic  processes.  Thus,  there 
may  be  a  mismatch  between  the  OPP  as  laid  out  in  doctrine  and  taught  at  training  and  education 
institutions  within  the  CF,  and  the  planning  process  as  practiced  by  command  teams  in  more 
operational  settings,  especially  at  the  Brigade  level  and  below. 

Specifically,  the  current  work  includes  the  development  of  an  alternative  planning  process  based 
on  intuitive  decision  making  (referred  to  as  the  Intuitive  Operations  Planning  Process  or  IOPP), 
the  development  of  a  training  course  for  the  IOPP,  and  an  evaluation  of  the  effectiveness  of  the 
IOPP  compared  to  the  existing  CF  OPP. 

The  first  iteration  of  the  IOPP  model  exhibits  the  ‘best’  characteristics  of  other  intuitive  planning 
models  (Kievenaar,  1997;  Schmitt  &  Klein,  1999;  Thunholm,  2005;  Whitehurst,  2002)  and 
incorporates  findings  from  previous  work  investigating  application  of  the  OPP  in  the  CF  (Bruyn 
et  al.,  2005),  while  maintaining  much  of  the  terminology,  outputs  generated  and  formal  staff 
briefings  used  in  the  OPP  in  order  to  promote  level  of  acceptance  by  CF  practitioners  and  face 
validity  of  the  IOPP.  The  key  characteristics  of  the  IOPP  are  as  follows: 

•  The  development  of  only  1  CO  A  at  a  time; 

•  Mental  wargaming  as  opposed  to  traditional  wargaming; 

•  Continuous  situational  awareness  throughout  the  process; 

•  Feedback  loops  from  COA  development,  COA  validation  and  Plan  Development  to  the 
beginning  of  Mission  Analysis; 

•  The  concurrent  performance  of  Mission  Analysis  and  COA  Development; 

•  The  continuous  reciprocal  input  and  feedback  from  Situation  Awareness  (SA)  to  every 
other  high  level  function  in  the  IOPP,  and, 

•  Early  and  frequent  communication  with  lower  formations. 

A  web-based  training  course  for  both  the  OPP  and  IOPP  was  developed  as  it  allows  more 
flexibility  in  terms  of  the  delivery  of  the  training  than  lecture-based  training.  Aside  from  some 
minor  technological  complications,  delivery  of  the  training  courses  was  successful  and  overall 
participants  provided  positive  feedback  on  the  content  and  format  of  both  the  OPP  and  IOPP 
training.  Participants  did  suggest,  however,  that  the  web-based  training  should  be  supplemented 
by  a  form  of  face-to-face  training  such  as  lecture-based  or  syndicate  group  work.  As  well, 
participants  were  dissatisfied  with  the  lack  of  opportunity  to  ask  questions  at  any  time  during  the 
training.  This  could  be  addressed  by  incorporating  some  form  of  face-to-face  interaction  into  the 
training.  Participant  feedback  on  the  training  also  suggests  that  incorporating  more  concrete 
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examples  of  how  to  apply  the  IOPP  and  what  products  should  be  generated,  would  have  made  the 
training  more  effective. 

The  experimental  evaluation  represented  the  first  application  of  the  IOPP  in  an  operational 
context.  In  general,  results  suggest  that  teams  were  able  to  follow  the  IOPP  without  much 
difficulty,  although  some  participants  were  somewhat  reluctant  to  commit  to  a  single  COA.  The 
experimental  evaluation  also  showed  that  the  IOPP  outperformed  OPP  in  terms  of  efficiency, 
number  of  steps  performed,  quality  of  planning  products,  usability  of  planning  process, 
operational  effectiveness  (suitability  in  a  variety  of  operational  contexts),  and  participant 
workload  levels  (i.e.  IOPP  leads  to  lower  workload).  It  was  also  observed  that  application  of  the 
IOPP  resulted  in  more  looping  and  repeated  steps  compared  to  the  OPP.  This  was  anticipated 
given  that  the  IOPP  was  designed  to  be  iterative  and  also  prescribes  the  way  in  which  iterations 
should  occur.  On  the  other  hand,  participants’  perceived  level  of  trust  and  reliability  was  lower 
for  the  IOPP  compared  to  the  OPP.  They  also  felt  that  the  IOPP  was  more  likely  to  lead  to 
errors  (and  more  critical  errors)  in  a  Plan. 

In  terms  of  participant  feedback  on  the  IOPP,  teams  noted  a  number  of  benefits  of  the  IOPP 
including  efficiency,  flexibility,  increased  collaboration,  earlier  identification  of  COA,  more 
detailed  analysis  of  COA,  earlier  and  more  frequent  communication  with  lower  formations,  and 
the  opportunity  to  take  advantage  of  the  experience  of  the  Commander.  Teams  also  noted  some 
potential  limitations  of  the  IOPP  including  the  risk  of  committing  to  a  bad  COA  too  soon, 
reliance  on  experience  and  strength  (in  terms  of  personality)  of  the  Commander,  fewer 
“benchmarks”  on  the  way  to  the  Plan,  suitability  in  an  asymmetric  environment,  and  the 
tendency  to  get  off  track  because  of  focus  on  situational  awareness. 

Limitations  of  the  study,  including  the  participants’  lack  of  experience  with  the  IOPP,  the  level  of 
realism  of  the  experiment,  the  presence  of  a  learning  effect,  and  challenges  associated  with 
observer  data  collection  are  addressed  and  recommendations  are  made  for  future  studies. 

Recommendations  for  future  work  are  presented  with  respect  to  the  IOPP  model,  training  of  the 
IOPP  and  future  experimental  evaluation.  In  terms  of  the  IOPP  model,  further  development  of 
the  model  should  be  pursued.  Publication  and  communication  of  the  IOPP  to  a  broader  audience 
is  essential  to  successful  development  of  further  iterations  of  the  IOPP.  In  communicating  the 
IOPP  to  a  broader  audience,  the  fact  that  the  IOPP  incorporates  the  best  aspects  of  existing 
intuitive  models  and  incorporates  continuous  SA  check  and  iterative  mission  analysis  and  COA 
development,  should  be  emphasized.  Future  research  should  specifically  address  criteria  for 
success  and  SA  in  general  in  terms  of  how  these  concepts  should  best  be  implemented  and 
represented  in  the  planning  process. 

Several  opportunities  exist  for  training  enhancements  including  the  development  of  the  SA 
component  of  training,  improved  online  capabilities  of  the  training,  the  provision  of  24/7 
support,  online  conferencing  with  audio  and  visual  capabilities,  the  use  of  virtual  syndicates,  and 
the  use  of  more  concrete  examples. 

Future  experimental  evaluations  should  comprise  at  least  a  3 -day  exercise  in  order  to  circumvent 
first  day  familiarization.  Also,  the  involvement  of  more  planning  teams  would  be  desirable  in 
order  to  create  a  totally  balanced  experiment.  The  involvement  of  more  participants  would  also 
increase  the  validity  and  generalizability  of  results.  A  more  representative  training  environment 
may  also  enhance  the  realism  of  the  experience.  In  addition,  more  dynamic  scenarios  that  better 
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represents  modem  warfare,  such  as  asymmetric  warfare,  would  be  desirable.  Further,  scenarios 
should  be  different  enough  that  there  is  no  learning  effect  between  the  scenarios.  That  is, 
planning  teams  should  have  to  do  an  equivalent  amount  of  mission  planning  for  both  scenarios. 
Teams  would  also  likely  benefit  from  more  detailed  examples  of  expected  output  (e.g.  battle 
matrix). 
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Sommaire 


Ce  travail  est  la  quatrieme  phase  d’un  projet  d’enquete  sur  le  processus  de  planification 
operationnelle  (PPO)  des  Forces  canadiennes  (FC)  et  un  processus  de  planification  de  rechange 
fonde  sur  la  prise  de  decisions  intuitives.  II  vise  a  appuyer  un  projet  plus  vaste,  le  projet 
Minerve,  qui  est  axe  sur  un  nouvel  examen  du  commandement  et  controle  (C2)  en  particulier  le 
PPO  FC  dans  la  force  terrestre,  compte  tenu  de  la  mise  en  place  des  systemes  C2  numerises.  Le 
PPO  FC  represente  un  processus  de  prise  de  decision  analytique  au  cours  duquel  1)  plusieurs 
solutions  au  probleme  doivent  etre  evaluees  et  la  meilleure  solution  choisie  et  2)  revaluation  de 
solutions  de  rechange  doit  etre  effectuee  par  une  comparaison  exhaustive  critere  par  critere.  Les 
recherches  en  sciences  cognitives  indiquent  qu’une  grande  partie  de  la  prise  de  decision  humaine 
est  intuitive,  c’est-a-dire  qu’elle  s’effectue  selon  un  processus  moins  analytique,  moins  formel. 
Par  consequent,  il  peut  y  avoir  une  discordance  entre  le  PPO  tel  que  decrit  dans  la  doctrine  et 
enseigne  dans  les  etablissements  d’education  et  d’instruction  des  FC  et  le  processus  de 
planification  tel  que  mis  en  pratique  par  l’equipe  de  commandement  dans  les  contextes  plus 
operationnels,  en  particulier  a  la  brigade  et  aux  niveaux  inferieurs. 

Plus  particulierement,  ce  travail  comprend  P  elaboration  d’un  processus  de  planification  de 
rechange  fonde  sur  la  prise  de  decisions  intuitives  (appele  Processus  de  planification  intuitive  des 
operations  ou  PPIO),  la  preparation  d’un  cours  sur  le  PPIO  et  une  evaluation  de  l’efficacite  du 
PPIO  par  rapport  au  PPO  FC  actuel. 


La  premiere  iteration  du  PPIO  presente  les  meilleures  caracteristiques  d’autres  modeles  de 
planification  intuitive  (Kievenaar,  1997;  Schmitt  &  Klein,  1999;  Thunholm,  2005;  Whitehurst, 
2002)  et  inclut  des  resultats  de  travaux  de  recherche  precedents  portant  sur  la  mise  en  oeuvre  du 
PPO  dans  les  FC  (Bruyn  et  coll.,  2005),  tout  en  conservant  une  grande  partie  de  la  terminologie 
du  PPO,  les  commentaires  generes  par  le  PPO  et  des  seances  d’information  officielles  employees 
dans  le  cadre  de  ce  processus  afin  de  favoriser  le  degre  d’acceptation  des  utilisateurs  des  FC  et  la 
validite  apparente  du  PPIO.  Les  principales  caracteristiques  du  PPIO  sont  les  suivantes  : 

•  L’elaboration  d’un  seul  plan  d’action  a  la  fois; 

•  Utilisation  de  jeux  de  guerre  mentaux  au  lieu  des  jeux  de  guerre  traditionnels; 

•  Connaissance  continue  de  la  situation  tout  au  long  du  processus; 

•  Boucles  de  retroaction  sur  P  elaboration  et  la  validation  de  plan  d’action,  et  preparation 
du  plan  des  le  debut  de  Paralyse  de  la  mission; 

•  Analyse  de  la  mission  et  elaboration  du  plan  d’action  executees  simultanement; 

•  Commentaires  continus  et  reciproques  de  la  connaissance  de  la  situation  (CS)  a  toutes  les 
autres  fonctions  de  niveau  superieur  du  PPIO; 

•  Communications  frequentes  des  le  debut  du  processus  avec  les  formations  de  niveau 
inferieur. 

Un  cours  en  ligne  sur  le  PPO  et  le  PPIO  a  ete  prepare  car  il  permet  une  plus  grande  souplesse 
concernant  la  prestation  de  l’instruction  qu’un  cours  en  salle  de  classe.  Outre  quelques 
complications  technologiques,  la  prestation  de  l’instruction  s’est  averee  un  succes.  L’ensemble 
des  participants  ont  fait  des  commentaires  positifs  sur  le  contenu  et  le  format  de  l’instruction  sur 
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le  PPO  et  le  PPIO.  Les  participants  ont  cependant  propose  de  remplacer  l’instruction  en  ligne 
par  une  instruction  traditionnelle  comme  des  exposes  ou  des  travaux  d’equipe  en  atelier.  Les 
participants  se  sont  aussi  montres  insatisfaits  du  peu  d’occasion  de  poser  des  questions  a  tout 
moment  au  cours  de  l’instruction.  Cette  situation  peut  etre  corrigee  en  ajoutant  des  interactions 
en  face-a-face.  Les  commentaires  des  participants  sur  l’instruction  laissent  aussi  penser  que 
l’ajout  d’exemples  plus  concrets  pour  demontrer  comment  appliquer  le  PPIO  et  les  produits 
attendus  aurait  permis  d’ameliorer  l’efficacite  de  1’ instruction. 

L’evaluation  experimentale  constitue  la  premiere  application  du  PPIO  dans  un  contexte 
operationnel.  De  maniere  generate,  les  resultats  laissent  penser  que  les  equipes  ont  ete  en  mesure 
de  suivre  le  PPIO  sans  trop  de  difficulte,  meme  si  certains  participants  semblaient  hesitants  a 
soumettre  un  seul  plan  d’action.  L’evaluation  experimentale  a  egalement  demontre  que  le  PPIO 
etait  superieur  au  PPO  en  ce  qui  a  trait  a  l’efficacite,  au  nombre  d’etapes  necessaires,  a  la  qualite 
des  produits  de  planification,  a  la  convivialite  du  processus  de  planification,  a  l’efficacite 
operationnelle  (pertinence  dans  une  variete  de  contextes  operationnels)  et  a  la  charge  de  travail 
des  participants  (c.-a-d.  que  le  PPIO  entraine  une  diminution  de  la  charge  de  travail).  On  a  aussi 
observe  que  l’application  du  PPIO  donnait  lieu  a  davantage  de  boucles  et  de  repetitions  d’etapes 
par  rapport  au  PPO.  On  avait  prevu  la  situation  etant  donne  que  le  PPIO  a  ete  concu  pour  etre 
iteratif  et  pour  determiner  de  quelle  faqon  les  iterations  devraient  se  produire.  D’autre  part,  les 
participants  ont  trouve  que  le  niveau  de  confiance  et  de  fiabilite  du  PPIO  etait  inferieur  a  celui  du 
PPO.  Ils  ont  aussi  eu  l’impression  que  le  PPIO  risquait  davantage  de  donner  lieu  a  des  erreurs  (et 
des  erreurs  plus  critiques)  dans  un  plan. 

En  ce  qui  a  trait  aux  commentaires  des  participants,  les  equipes  ont  fait  remarquer  que  le  PPIO 
comportait  un  certain  nombre  d’avantages,  notamment  l’efficacite,  la  souplesse,  une 
collaboration  accrue,  la  determination  hative  d’un  plan  d’action,  une  analyse  plus  detaillee  du 
plan  d’action,  des  communications  frequentes  et  plus  rapides  avec  les  formations  de  niveau 
inferieur  et  l’occasion  de  tirer  parti  de  l’experience  du  commandant.  Les  equipes  ont  aussi 
remarque  que  le  PPIO  presentait  des  limites  potentielles,  notamment  le  risque  de  soumettre  un 
mauvais  plan  d’action  trop  hativement,  la  confiance  accordee  a  l’experience  et  aux  forces  (en 
parlant  de  la  personnalite)  du  commandant,  des  reperes  moins  nombreux  entourant  la  preparation 
du  plan,  la  pertinence  d’un  environnement  asymetrique  et  la  tendance  a  se  desorienter  en  raison 
de  l’importance  accordee  a  la  connaissance  de  la  situation. 

La  question  entourant  les  limites  de  1’ etude  est  actuellement  examinee,  notamment  le  manque 
d’experience  des  participants  avec  le  PPIO,  le  degre  de  realisme  de  l’experience,  la  presence 
d’un  effet  d’apprentissage  et  les  defis  associes  a  la  collecte  de  donnees  par  l’observateur  et  des 
recommandations  sont  faites  pour  les  etudes  subsequentes. 

Des  recommandations  sur  les  travaux  a  venir  concernant  le  modele  PPIO,  l’instruction  sur  le 
PPIO  et  la  prochaine  evaluation  experimentale  sont  presentees.  En  ce  qui  concerne  le  modele 
PPIO,  il  faut  continuer  a  le  developper.  La  publication  et  la  transmission  du  PPIO  a  un  plus 
grand  nombre  de  personnes  est  essentiel  pour  reussir  a  preparer  d’autres  iterations  du  PPIO.  En 
communiquant  le  PPIO  a  un  plus  grand  nombre  de  personnes,  il  faudrait  insister  sur  le  fait  qu’il 
comprend  les  meilleurs  aspects  des  modeles  intuitifs  existants  et  inclut  une  verification  continue 
de  la  CS,  une  analyse  iterative  de  mission  et  la  preparation  d’un  plan  d’action.  Les  prochaines 
recherches  devraient  s’attarder  plus  particulierement  aux  criteres  de  succes  et  a  la  CS  en  general 
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en  ce  qui  concerne  la  meilleure  faqon  de  mettre  en  oeuvre  ces  concepts  et  de  les  presenter  dans  le 
processus  de  planification. 

II  existe  plusieurs  possibilites  pour  ameliorer  1’ instruction,  notamment  preparer  1’ instruction  sur 
la  CS,  ameliorer  les  capacites  de  l’instruction  en  ligne,  fournir  un  soutien  24  heures  par  jour,  7 
jours  par  semaine,  tenir  des  conferences  en  ligne  avec  image  et  son,  utiliser  des  ateliers  virtuels 
et  utiliser  des  exemples  plus  concrets. 

Les  prochaines  evaluations  experimental  devraient  inclure  au  moins  un  exercice  de  trois  jours 
afin  d’eviter  la  premiere  journee  de  familiarisation.  De  plus,  il  serait  souhaitable  qu’un  plus 
grand  nombre  d’equipes  de  planification  participent  afin  de  creer  une  experience  totalement 
equilibree.  La  participation  d’un  plus  grand  nombre  de  personnes  augmenterait  aussi  la  validite  et 
la  generalisabilite  des  resultats.  Un  environnement  destruction  plus  representatif  permettrait 
aussi  de  favoriser  le  realisme  de  l’experience.  En  outre,  il  serait  indique  de  disposer  de  scenarios 
plus  dynamiques  et  plus  representatifs  de  la  guerre  moderne,  comme  la  guerre  asymetrique  par 
exemple.  Les  scenarios  devraient  egalement  etre  assez  differents  pour  eviter  l’effet 
d’apprentissage  entre  les  scenarios.  C’est-a-dire  que  les  equipes  de  planification  devraient  faire 
autant  de  travail  de  planification  de  mission  pour  les  deux  scenarios.  Les  equipes  tireraient 
probablement  davantage  profit  d’exemples  plus  detailles  des  resultats  attendus  (p.  ex.  matrice  de 
combat). 
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1.  Introduction 


1.1  Background 

The  intention  of  Project  Minerva  is  to  re-examine  Land  Force  Command  and  Control  (C2)  in 
light  of  the  implementation  of  digitized  C2  systems  within  the  context  of  the  Athene  Tactical 
System.  The  Land  Force  wishes  to  develop  procedures  that  capitalize  on  the  strengths  of 
digitization.  Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  supporting  Project 
Minerva  by  carrying  out  studies  into  human  decision  making,  planning  and  the  Operations 
Planning  Process  (OPP). 

Project  Minerva  focuses  on  the  OPP,  which  is  the  prescribed  method  of  planning  for  a  mission. 
Although  the  OPP  was  developed  without  any  explicit  relation  to  psychological  theories  of 
problem  solving  and  decision  making,  it  is  consistent  with  analytic  decision  making  (Bryant, 
Webb,  &  McCann,  2003).  In  particular,  the  OPP  conforms  to  two  major  premises  of  analytic 
decision  making;  1)  multiple  solutions  to  the  problem  must  be  evaluated  and  the  best  selected, 
and  2)  evaluation  of  solution  alternatives  must  be  performed  through  exhaustive  factor-by-factor 
comparison. 

Research  in  the  cognitive  sciences  has  suggested  that  a  large  portion  of  human  decision  making  is 
conducted  intuitively;  i.e.  by  less  formal,  non-analytic  processes.  Thus,  there  may  be  a 
mismatch  between  the  OPP  as  laid  out  in  doctrine  and  taught  at  training  and  education  institutions 
within  the  Canadian  Forces  (CF),  and  the  planning  process  as  practiced  by  command  teams  in  the 
field.  In  particular,  command  teams  at  Brigade  level  and  below  may  engage  in  a  more  intuitive 
process  than  the  doctrinal  OPP.  An  intuitive  planning  process  may  be  preferable  to  an  analytic 
process  as  intuitive  reasoning  has  been  demonstrated  to  require  less  information  and  consume  less 
time  than  strictly  analytic  processes.  Even  where  analytic  processes  have  advantages,  innate 
tendencies  of  humans  to  think  intuitively  may  reduce  the  effectiveness  of  an  analytic  procedure 
such  as  the  OPP  when  put  into  practice.  These  arguments  for  intuitive  procedures,  however, 
may  not  apply  when  decision  making  is  considered  in  the  context  of  highly  complex,  dynamic 
problem  scenarios  involving  many  different  planning  participants. 

The  current  work  represents  the  fourth  phase  of  this  project  and  includes  the  development  of  an 
alternative  planning  process  based  on  intuitive  decision  making  and  an  evaluation  of  the 
effectiveness  of  the  intuitive-based  planning  process  compared  to  the  existing  OPP.  The 
following  section  briefly  describes  work  performed  in  the  first  three  phases  of  the  project. 

The  project  has  been  contracted  to  Human systems  Incorporated®  (Humansystems®,  HSI®)  for 
DRDC  Toronto  under  contract  W7711-0047907/001/TOR.  The  Scientific  Authority  for  this  work 
is  Dr.  David  Bryant. 

1 .2  Previous  Work 

Three  studies  have  been  conducted  by  DRDC  Toronto  under  Project  Minerva:  a  function  flow 
analysis  of  the  CF  OPP  as  described  in  doctrine  (Bruyn  et  al,  2004);  a  literature  survey  for 
papers  concerning  intuitive  decision  making  and  planning  (Lamoureux,  2004);  and  a  function 
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flow  analysis  and  comparison  of  the  CF  OPP  in  practice  with  the  CF  OPP  in  doctrine  (Bruyn  et 
al,  2005). 

The  first  study  (Bruyn  et  al,  2004)  determined  that  the  CF  OPP  was  slightly  different  to  the  OPP 
practiced  by  the  Land  Force  in  that  it  had  five  steps  instead  of  six.  The  CFOPP  is  the 
operational  level  planning  process  outlined  in  Joint  Doctrine  (National  Defence,  2002)  and  is  the 
planning  process  taught  at  the  Canadian  Forces  College  (CFC)  in  Toronto.  The  OPP,  on  the 
other  hand,  exists  in  the  Land  Force  Command  doctrine  (National  Defence,  1996)  and  is  the 
planning  process  taught  at  Canadian  Land  Force  Command  and  Staff  College  (CLFCSC)  in 
Kingston. 

Figure  1  shows  the  six  step  process  followed  by  the  Land  Force.  Observations  were  also  made 
regarding  the  employment  of  the  OPP  by  Brigade-level  Staffs  and  below.  The  OPP  was 
described  in  relation  to  the  different  levels  of  command  and  the  other  approaches  to  planning 
(notably  the  Estimate).  Finally,  all  the  steps  in  the  OPP  were  described  in  terms  of  their 
triggers,  their  outputs,  their  information  requirements,  and  the  Staff  responsible  for  them. 


Figure  1 :  Six  step  planning  process  used  by  the  Land  Force 

The  second  study  (Lamoureux,  2004)  surveyed  the  available  literature  regarding  decision  making 
and  planning,  paying  particular  attention  to  literature  that  specifically  addressed  intuitive  decision 
making  and  planning.  Although  this  work  was  not  a  literature  review  per  se  (it  sought  only  to 
identify  and  obtain  the  most  relevant  literature  and  draw  some  high  level  conclusions  from  the 
abstracts),  the  conclusions  drawn  provide  a  preliminary  appreciation  of  the  differences  between 
intuitive  and  analytic  approaches  to  planning. 

The  third  study  (Bruyn  et  al,  2005)  involved  the  observation  of  a  Brigade  planning  Staff  during 
Exercise  VIRTUAL  RAM  at  Canadian  Forces  Base  (CFB)  Edmonton.  This  observation  was 
conducted  with  1  Canadian  Mechanized  Brigade  Group  (1  CMBG)  in  realistic  field  conditions  in 
January  2005.  Several  conclusions  from  this  study  are  particularly  noteworthy  in  the  context  of 
the  current  contract.  First,  it  was  apparent  that  the  role  of  the  Commander  (Comd)  in  the 
application  of  the  OPP  was  extremely  significant.  During  this  study,  the  Commander  conducted 
several  steps  of  the  OPP  himself,  provided  detailed  guidance  regarding  how  to  apply  the  OPP, 
and  also  provided,  during  one  planning  cycle,  the  specific  COA  to  be  developed.  This  approach 
to  planning  by  the  Commander  can  be  considered  intuitive,  based  on  his  experience  and  training. 
A  second  observation  was  that,  independent  of  the  Commander’s  guidance,  the  planning  Staff 
only  engaged  in  some  of  the  OPP  steps,  skipping  over  many  that  may  or  may  not  have  been 
relevant.  Related  to  this,  it  was  apparent  that  the  Staff  were  ‘looping’  back  and  forth  between 
different  steps  in  the  OPP,  rather  than  following  the  OPP  sequentially.  Figure  2  shows  the  type 
of  ‘looping’  seen  between  steps  of  the  OPP  (steps  are  in  alphabetical  order).  This  implies  an 
iterative  process  of  COA  and  Plan  Development,  from  which  one  can  deduce  that  planning 
behaviour  is  more  opportunistic  in  terms  of  the  factors  considered,  which  is  closer  in  approach  to 
intuitive  decision  making  than  analytic.  Finally,  it  was  felt  that  the  planning  Staff  did  not  have 
sufficient  time  or  resources  to  conscientiously  apply  the  OPP  as  prescribed  in  doctrine.  While  at 
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a  high  level  (i.e.  the  highest  level  functions  such  as  1.0  Initiation,  2.0  Orientation)  the  Bde  Staff, 
in  general,  followed  the  sequence  of  the  OPP,  at  a  low  level  (i.e.  low  level  functions  such  as 
‘Assess  suitability  of  each  COA’)  they  were  observed  to  abbreviate  steps,  move  backward  and 
forward  between  steps,  and  to  not  always  engage  in  the  exhaustive  factor-by -factor  consideration 
of  options.  This  suggests  the  use  of  intuitive  strategies  by  individuals  in  the  Plans  Cell. 


Start 


Figure  2:  Example  of  function  flow,  showing  actual  flow  between  doctrinal 

functions 

Bruyn  et  al.  (2005)  made  several  recommendations  for  follow-on  work,  including  the 
development  of  intuitive  models  of  planning,  the  creation  of  a  training  syllabus,  and  the 
comparative  evaluation  of  analytic  and  intuitive  planning  approaches  at  the  operational  level. 


1.3  Current  Work 


The  overall  objective  of  this  phase  of  the  project  was  to  develop  an  intuitive  planning  process 
(referred  to  as  the  Intuitive  Operations  Planning  Process,  or  IOPP),  create  a  training  program  to 
teach  the  planning  process,  and  evaluate  the  effectiveness  of  a  planning  process  based  on  intuitive 
planning  and  decision  making  relative  to  the  existing  OPP,  which  is  consistent  with  what  has 
been  termed  analytic  decision  making  (Bryant,  Webb,  and  McCann,  2003).  To  compare  the 
effectiveness  of  the  IOPP  and  OPP  an  experiment  was  performed  to  assess  the  effectiveness  and 
efficiency  of  each  process  in  a  simulated  battle  group  level  exercise. 

The  following  work  tasks  were  performed  for  this  project  (task  numbers  correspond  to  tasks  as 
outlined  in  the  Statement  of  Work  (SOW): 


•  Task  #1:  Project  Management 

•  Task  #2:  Develop  and  Document  Intuitive/Abbreviated  Planning  Process 

•  Task  #3:  Develop  Training  Course  for  Battalion/Brigade  Staffs 

•  Task  #4:  Design  and  Conduct  Experimental  Evaluation  of  Intuitive  Planning  Process 

•  Task  #5:  Prepare  and  Submit  Final  Report 
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This  document  represents  the  final  report  for  the  project  and  therefore  includes  a  comprehensive 
description  of  Tasks  2-4. 
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2.  Development  and  Documentation  of 
Intuitive  Planning  Process 

2.1  Method 

2.1 .1  Review  of  existing  planning  models 

The  development  of  an  alternative  planning  process  based  on  intuitive  decision  making  and  planning 
began  with  a  review  of  existing  or  other  proposed  military  intuitive  planning  models.  The  models 
reviewed  included  The  Recognition  Planning  Model  (Schmitt  &  Klein,  1999),  The  Planning  Under 
Time  Pressure  (PUT)  model  (Thunholm,  2005),  the  Accelerated  Decision-Making  Model 
(Kievennar,  1997),  and  the  Abbreviated  Military  Planning  Process  (AMPP)  (Whitehurst,  2002).  In 
reviewing  the  models,  characteristics  of  the  models  were  compared  and  contrasted.  Table  1  shows 
the  comparison  of  advantages  and  disadvantages  of  the  various  intuitive  planning  models  reviewed. 
From  this,  we  created  a  “wish  list”  that  included  the  “best”  elements  from  each  model.  No  strict 
definition  of  “best”  was  used  during  this  work.  Rather,  it  was  assumed  that  what  was  observed 
during  the  third  study  represented  the  manner  in  which  individuals  would  choose  to  plan.  Thus,  the 
“best”  elements  of  the  various  intuitive  planning  models  were  those  that  were  common  to  two  or 
more  models  and  those  that  reflected  activities  and  behaviours  previously  observed  in  planning 
Staffs  in  operational  settings  (Bruyn  et  al.,  2005). 

Table  1:  Advantages  and  disadvantages  of  intuitive  planning  models  reviewed 


Model  Reviewed 

Advantages 

Disadvantages 

RPM  (Schmitt  &  Klein,  1999) 

Many  feedback  loops 

Situation  awareness  not  part  of 
planning  process 

No  development  of  ECOA 

Wargaming  takes  form  of  mental 
simulation  by  which  coordination 
issues  are  resolved  and  contingencies 
are  developed 

No  description  of  sub-functions 

Model  shows  embedded  boxes 
showing  that  steps  aren’t  mutually 
exclusive 

Model  shows  time  process  along  a 
time  scale 

PUT  (Thunholm,  2005) 

Situation  assessment  included  as  part 
of  planning  process 

The  reasoning  behind  understanding 
mission  before  situation  assessment 

Is  questionable 

Defining  explicit  criteria  of  success 
(e.g.  element  of  surprise,  local 

Redundant  numbering  is  confusing 
(i.e.  Step  2,  sub-step  2) 
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Model  Reviewed 

Advantages 

Disadvantages 

superiority) 

Provides  a  list  of  outputs 

Model  is  flexible  and  prescriptive  in 
that  it  states  how  steps  that  can  be 
excluded  in  situations  of  extreme 
time-pressure 

Accelerated  Decision-Making  Model 
(Kievennar,  1997) 

Increased  communication  with  lower 
formations  -  4  warning  orders  issued 

Mission  analysis  separate  from  COA 
development 

Outlines  some  lower  level  functions 

Wargaming  separate  from  COA 
development 

Linear  process 

Abbreviated  Military  Planning  Process 
(Whitehurst,  2002) 

Eliiminates  COA  comparison  and 
approval  steps  as  only  1  COA  is 
developed  by  Comd  (therefore  more 
efficient  process) 

Intention  of  wargaming  is  not  to 
synchronize  COA 

Critical  thinking  (i.e.  “crystal  ball”) 
technique  used  to  identify 
assumptions,  gaps  in  mental  model 
and  contingency  planning  (i.e. 
branches  &  sequels) 

Defines  lower  level  functions 

Considers  group  decision  making 
biases  (e.g.  groupthink) 

Provides  explicit  steps  to  develop 
enemy  COA  (ECOA) 

The  following  features  were  identified  as  being  common  to  all  intuitive  planning  models 
reviewed: 

1.  There  is  only  1  CO  A  developed; 

2.  The  Commander  selects  a  single  COA  based  on  experience;  and, 

3.  Wargaming  is  used  for  synchronization  rather  than  comparison/evaluation  of  COAs. 

It  was  therefore  determined  that  these  features  should  be  incorporated  into  our  intuitive  planning 
model  and  were  therefore  added  to  our  wish  list. 

Previous  work  for  this  project  (Bruyn,  Rehak,  Lamoureux  &  Vokac,  2005)  was  also  reviewed  to 
determine  general  pattern  of  functions  performed  by  Brigade-  level  planning  teams  in  the  context 
of  an  operational  exercise.  This  information  was  added  to  the  overall  wish  list  of  elements  to  be 
included  in  the  intuitive  planning  models. 

Based  on  the  review  of  existing  or  proposed  intuitive  planning  models  and  the  results  of  previous 
work  for  this  contract,  the  following  “wish  list”  for  the  intuitive  planning  process  was  created: 
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•  Skeleton  COA  and  mission  analysis  simultaneously; 

•  Criteria  for  success; 

•  Emphasize  steps  that  can  be  excluded; 

•  Situation  assessment  in  planning  process; 

•  Steps  are  not  mutually  exclusive; 

•  Model  should  include  timescale; 

•  ECOA  development; 

•  Feedback  loops; 

•  Description  of  sub-functions; 

•  More  frequent  Warning  Order  (or  other  form  of  communication)  to  lower 
formations; 

•  Make  it  specific  or  adaptable  to  current  CF  operation/mission  types; 

•  Identify  assumptions,  gaps  in  mental  model,  branches  and  sequels; 

•  Comd  present  for  mission  analysis  &  COA  development; 

•  Make  explicit  who  replaces  Comd  if  absent  (i.e.  Chief  of  Staff  (COS)); 

•  Design  it  so  it  is  apparent  that  it  is  not  a  step-wise,  linear  process;  and 

•  Want  to  keep  terminology  consistent  with  OPP. 

In  addition  to  the  wish  list,  there  was  a  desire  to  promote  the  level  of  acceptance  and  face  validity 
of  the  new  intuitive  planning  model  by: 

•  Maintaining  terminology  used  in  OPP  &  CFOPP  (e.g.  Comd’s  intent,  wargaming, 
mission  analysis,  COA  development); 

•  Maintaining  outputs  generated  (e.g.  mission  statement); 

•  Maintaining  planning  tools  used; 

•  Ensuring  that  staff  still  has  meaningful  role  in  planning  process  (i.e.  Comd  not  doing 
everything);  and 

•  Ensuring  realistic  requirements  on  Comd’s  /availability  time  (i.e.  Comd’s  presence 
not  required  for  all  steps). 

2.2  Results 

2.2.1  Creation  of  overall  model 

A  model  of  the  overall  IOPP  with  the  highest  level  functions  was  created  to  incorporate  as  many 
elements  from  the  wish  list  as  possible.  There  were  several  iterations  based  on  feedback  from 
project  members,  the  Scientific  Authority  and  a  retired  military  Subject  Matter  Expert  (SME) 
who  was  well  versed  in  the  OPP. 

The  final  overall  model  of  the  OPP,  showing  the  highest  level  steps  only,  is  shown  in  Figure  3. 
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Figure  3:  Intuitive  Operational  Planning  Process  (IOPP) 

The  model  consists  of  5  steps,  each  of  which  comprises  several  sub-steps  which  are  discussed 
below.  The  five  high  level  steps  include: 

1.  Situational  Awareness  (S A); 

2.  Mission  analysis; 

3.  COA  Development; 

4.  COA  Validation;  and 

5.  Plan  Development. 

The  input  or  trigger  to  initiate  the  IOPP  is  receipt  of  a  mission  from  higher  command.  The 
output  of  the  IOPP  is  the  issue  of  orders  to  lower  command. 

Key  features  of  the  overall  IOPP  model  include: 

•  Feedback  loops  from  COA  development,  COA  validation  and  Plan  Development  to  the 
beginning  of  Mission  Analysis; 

•  The  concurrent  performance  of  Mission  Analysis  and  COA  Development  (shown  by  the 
adjacent  triangles  as  well  as  the  double  arrows  between  the  two  processes); 

•  The  continuous  reciprocal  input  and  feedback  from  SA  to  every  other  high  level  function  in 
the  IOPP;  and 

•  A  time  scale  across  the  bottom  to  show  functions  against  the  progression  of  time,  with  the 
proportions  of  each  step  representing  the  approximate  amount  of  time  that  it  is  anticipated 
that  a  planning  team  will  spend  on  each  step. 

The  continuous  reciprocal  input  to,  and  feedback  from,  SA  to  every  other  high  level  function  in 
the  IOPP  is  a  significant  feature  of  the  IOPP.  The  overall  IOPP  model  illustrates  that  a  certain 
amount  of  SA  must  be  attained  prior  to  the  start  of  mission  analysis.  As  the  Commander  and 
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Staff  work  through  the  planning  process,  they  increase  their  SA  (i.e.  awareness  of  enemy 
situation,  own  situation,  battlespace  environment,  other  military  and  civilian  resources)  and  this 
feeds  directly  into  the  development  and  refinement  of  the  COA.  Conversely,  it  is  highly 
probable  there  will  be  factors  identified  during  mission  analysis,  COA  development,  COA 
Validation  and  Plan  Development  that  help  to  build  the  SA  of  the  Commander  and  the  Staff. 

The  feedback  loops  along  the  bottom  of  the  model  were  included  primarily  because  in  the  IOPP  a 
single  COA  is  developed  at  one  time.  Therefore,  it  is  crucial  that  this  COA  is  tested  and 
validated  thoroughly  before  it  is  accepted  to  be  developed  into  a  CONOPS  (Concept  of 
Operations)  and  finally  Orders.  It  must  therefore  be  emphasized  that,  at  any  point  in  the  IOPP,  if 
it  becomes  clear  that  there  are  any  weaknesses,  limitations  or  ambiguity  associated  with  the  COA 
that  is  being  developed,  it  can  either  be  subjected  to  mission  analysis  again  or  another  COA  can 
be  selected  and  developed. 

The  concurrent  performance  of  mission  analysis  and  COA  development  was  intentional  as  it  was 
felt  that  the  initial  identification  of  a  COA  by  a  Commander  would  likely  occur  during  the  review 
of  own  and  higher  factors,  deductions,  assumptions,  assigned  tasks  and  objectives.  As  mission 
analysis  continues  and  Staff  define  the  most  likely  enemy  COA,  it  seems  logical  that  the  COA 
would  become  more  developed  and  refined  in  the  minds  of  the  Commander  and  Staff. 

The  next  step  was  to  decompose  the  high  level  functions  of  the  IOPP  into  a  number  of  lower 
level  functions.  The  exact  lower  level  functions  that  were  subsumed  within  each  higher  level 
function  was  based  on  previous  work,  knowledge  of  what  planning  products  of  the  existing  OPP 
should  remain,  review  with  an  SME  and  consideration  of  typical  Canadian  operations  and 
missions.  The  lower  level  functions  were  divided  into  functions  performed  by  Higher  Command, 
the  Commander,  the  Staff  and  Lower  Formations  to  show  the  interaction  between  the  players 
involved  in  the  planning  process.  These  interactions  can  be  seen  in  the  final  model  shown  in 
Annex  A. 

Highlights  of  this  model  include: 

•  The  development  of  only  1  COA  at  a  time; 

•  Mental  wargaming  as  opposed  to  traditional  wargaming;  and 

•  Continuous  situational  awareness  throughout  the  process. 

The  model  also  included  a  number  of  accompanying  notes  to  provide  further  explanation  for  each 
of  the  steps  within  the  process.  These  notes  can  be  seen  in  the  final  model  shown  in  Annex  A. 

The  model  was  initially  designed  with  the  intent  that  the  process  would  be  depicted  as  something 
other  than  a  step-wise  linear  process  like  the  OPP.  However,  this  proved  more  difficult  than 
initially  thought.  It  was  quickly  realized  that  creating  a  process  in  enough  detail  to  be  able  to 
train  individuals  in  it  requires  the  process  to  be  presented  in  a  systematic  fashion. 

Similar  to  the  overall  IOPP  model,  the  model  of  the  lower  level  functions  within  SA,  COA 
Development,  COA  Validation  and  Plan  Development  underwent  several  iterations  based  on 
feedback  from  an  SME,  the  Scientific  Authority  and  other  project  team  members.  As  previously 
noted,  the  final  version  IOPP  model  showing  the  lower  level  functions  is  included  in  Annex  A. 
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3.  Development  of  Training  Course  for 
Btn/Bde  Staff 


3.1  Determining  overall  requirements  for  training  courses 

3.1.1  Format 

To  allow  distance  education  and  flexibility  in  the  delivery  of  training,  a  web-based  course  was 
developed  for  the  IOPP  and  OPP.  This  course  was  not  be  location  or  time  dependent  for  the 
participants;  they  could  take  the  training  at  their  own  pace  and  at  their  choice  of  location.  Of 
course,  the  distance  learning  format  placed  certain  constraints  on  the  way  in  which  the  training 
material  was  presented.  All  content,  for  example,  had  to  be  contained  in  the  training  itself  as 
there  would  be  no  lecturer  present  to  expand  on  any  of  the  points  made  in  the  training.  It  was 
also  necessary  to  create  a  means  to  track  participation  in  the  training  to  ensure  that  it  was 
completed  by  participants.  Participation  was  monitored  in  two  ways.  First,  the  participants  were 
required  to  login  to  the  training  website,  and  certain  web  statistics  such  as  login  ID,  total  time 
logged  in,  slides  viewed,  etc.,  were  calculated.  Second,  quizzes  were  placed  at  the  end  of  each 
training  module.  Participant  answers  to  the  quiz  questions  were  tracked  to  ensure  that  they 
successfully  completed  each  quiz. 

An  administrative  email  address  was  created  to  provide  support  in  the  event  that  participants 
encountered  difficulties.  Participants  could  send  email  at  any  time  to  report  difficulties  or  ask 
questions  about  the  training.  The  support  staff  responded  to  emails  in  a  timely  manner. 

To  ensure  the  equitable  comparison  of  the  IOPP  and  OPP,  two  equivalent  training  programs  were 
developed:  one  for  the  OPP  and  one  for  the  IOPP.  Although  it  was  assumed  (and  a  requirement 
when  sourcing  participants)  that  participants  would  have  experience  with  the  OPP,  some 
refresher  training  was  desirable.  To  increase  the  likelihood  that  participants  would  approach  the 
OPP  and  IOPP  with  similar  levels  of  understanding,  the  IOPP  training  was  of  a  degree  of 
comprehensiveness  similar  to  that  of  the  OPP.  Participants  were  expected  to  have  greater 
experience  with  the  OPP  because  it  is  the  doctrinal  process  of  the  CF. 

Many  examples  were  incorporated  in  the  training  materials  to  make  the  training  easy  to 
assimilate  and  relevant  to  the  participants.  However,  time  and  resource  constraints  precluded  the 
development  of  an  example  of  sufficient  scope  that  it  could  be  developed  through  the  course  of 
the  training  to  illustrate  all  lessons.  Instead,  unrelated  examples  were  implemented  to  provide 
context  to  the  lessons  being  learned. 

The  training  was  created  in  PowerPoint  mainly  for  ease  of  creation,  but  also  because  it  could 
easily  be  incorporated  into  and  downloaded  from  a  website.  Also,  it  was  determined  that  there 
was  no  requirement  for  animation  so  a  format  such  as  Flash  was  not  required. 

The  training  was  designed  with  the  consideration  that  there  would  not  be  an  instructor  present  to 
provide  context  or  refer  back  to  the  main  objectives  or  points  during  the  course  of  the  training. 
That  is,  in  the  introductory  slides  of  both  training  courses  it  was  stated  clearly  why  the  training 
was  developed  and  what  the  purpose  of  the  training  was.  Second,  at  the  end  of  each  training 
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module,  a  review  is  provided  of  what  the  participants  have  learned  up  to  that  point  in  the 
training. 


3.1.2  Usability 


The  overall  presentation  of  the  training  was  designed  according  to  general  usability  principles. 
The  training  was  divided  into  a  number  of  individual  modules  so  that  participants  could  complete 
the  training  one  module  at  a  time  and  there  were  a  number  of  logical  places  to  take  a  break  and 
then  return  to  the  training.  First,  the  ability  of  the  participant  to  see  where  he  is  in  the  overall 
training  was  incorporated  (see  Figure  4  and  Figure  5).  Figure  4  shows  the  way  in  which  each 
step  within  the  process  was  linked  back  to  the  overall  model  (i.e.  highest  level  functions).  This 
was  shown  at  the  beginning  of  each  module  within  the  training.  Figure  5  shows  the  way  in 
which  the  participant  could  identify  which  specific  sub-step  within  each  step  (in  this  case  Mission 
Analysis)  was  being  discussed  in  the  training.  Each  sub-step  was  identified  in  a  similar  manner 
(i.e.  with  a  red  circle). 


Figure  4:  IOPP  Training  showing  both  detailed  process  and  overview 

As  noted  above,  the  online  format  provided  access  to  help  at  any  time  through  the  ability  to  email 
the  training  administrator.  The  training  also  ensured  the  repetition  of  the  main  points;  training 
modules  included  an  overview  page,  a  summary  page  and  quiz  questions  addressing  the  most 
important  points  of  each  module.  This  repetition  also  aided  in  memory  retention.  Finally, 
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general  readability  and  presentation  (i.e.  colour  coordination)  was  considered  in  the  expectation 
that  participants  would  undertake  the  training  on  screen  rather  than  as  printed  material. 


Mission  Statement 
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Staff 
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•Mission  statement  should  include  ‘who,  what,  when,  where,  why’ 

•Mission  Statement  includes  task,  purpose,  and  unifying 
statement.  May  include  limitations  and  constraints 

•Must  decide  on  relevant  deductions  reached  during  mission 
analysis  and  information  required  to  prepare  CPG  mission  task 
verbs  (e.g.  Secure,  Clear) 


Figure  5:  IOPP  showing  location  within  Mission  Analysis  step 

3.2  Creating  content  for  OPP  training 

Initially  it  was  intended  that  the  OPP  training  would  be  a  refresher  course  and  therefore  not 
provide  as  much  detail  as  the  IOPP  training  course.  However,  in  the  end  the  level  of  detail 
provided  in  both  the  OPP  and  IOPP  training  was  comparable.  That  is,  the  inputs,  processes  and 
outputs  described  for  individual  steps  of  the  OPP  and  IOPP  were  at  a  similar  level  of  detail. 

The  content  for  the  OPP  training  was  adapted  from  a  tabular  task  analysis  of  the  OPP  (Bruyn, 
Lamoureux  and  Vokac,  2004).  Within  the  tabular  task  analysis,  the  goals,  key  decisions,  outputs 
and  comments  for  each  step  of  the  process  were  incorporated  into  the  training.  The  original 
function  flow  diagrams  (Bruyn  et  al.,  2004)  were  modified  so  that  they  could  be  included  on  the 
slides. 
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Based  on  the  tabular  task  analysis,  it  was  determined  that  the  following  modules  would  be 
contained  within  the  OPP  training: 

•  Introduction/background; 

•  Overall  summary  of  OPP; 

•  Initiation; 

•  Orientation; 

•  COA  development; 

•  Plan  development; 

•  Plan  Review; 

•  Conclusion/ summary;  and 

•  Overall  summary  questions. 

3.2.1  Validation 

The  content  of  the  OPP  training,  along  with  all  supporting  documentation,  was  validated  by  an 
SME  who  is  well-versed  in  the  OPP.  This  SME  is  also  one  of  the  authors  of  this  paper. 

3.3  Creating  content  for  IOPP  training 

The  content  for  the  IOPP  training  was  adapted  from  the  model  itself  including  the  accompanying 
notes.  The  first  step  was  to  create  a  PowerPoint  presentation  from  the  initial  Flowcharter 
model.  Individual  modules  within  the  training  were  created  according  to  the  steps  within  the 
IOPP  model.  Points  on  the  slides  were  initially  created  from  the  notes  that  were  created  to 
accompany  the  model.  Additional  content  was  added  until  it  was  felt  that  there  was  a  sufficient 
level  of  detail  or  explanation  for  each  step  within  the  process.  Again,  the  fact  that  the  training 
was  designed  to  be  standalone  (i.e.  with  no  lecturer)  was  taken  into  account  when  determining 
the  level  of  detail  or  explanation  required  for  each  step  of  the  process. 

Based  on  the  IOPP  module,  it  was  determined  that  the  following  modules  would  be  contained 
within  the  OPP  training: 

•  Introduction/background; 

•  Overall  summary  of  IOPP; 

•  Situational  Awareness; 

•  Mission  Analysis; 

•  COA  Development; 

•  COA  Validation; 

•  Plan  Development; 

•  Conclusion/Summary;  and 

•  Overall  Feedback  Questionnaire. 
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As  part  of  the  development  of  the  IOPP,  it  was  recognised  that  there  are  certain  outputs  or 
products  of  a  planning  process  that  are  required  to  be  passed  down  to  subordinate  units.  Thus,  a 
few  of  the  products  produced  by  the  OPP  were  retained  in  the  IOPP  such  as  Initial  Intent, 
CONOPS  and  final  Plan  or  Orders.  These  products  were  maintained  in  order  to  keep  the  type  of 
information  sent  to  lower  formations  consistent  with  the  OPP,  thereby  not  disrupting  the  planning 
process  of  lower  formations. 

It  became  apparent  during  the  creation  of  the  training  that  difficulties  exist  in  the  creation  of  a 
training  course  for  an  intuitive  planning  process.  Initially,  a  goal  of  the  IOPP  was  to  keep  it  less 
systematic  than  the  OPP  in  keeping  with  the  less  structured  format  of  intuitive  processes. 
However,  because  the  IOPP  is  a  new  process  that  must  be  taught  and  certain  products  of  the  OPP 
could  not  be  eliminated,  the  IOPP  as  outlined  in  the  training  turned  out  to  be  a  somewhat 
systematic  process  itself.  Also,  it  was  thought  that  perhaps  the  IOPP  would  have  more  face 
validity  if  it  was  not  a  complete  departure  in  every  respect  from  the  planning  process  that  is 
currently  in  existence.  It  is  postulated  that,  although  initial  training  in  the  IOPP  needed  to  be 
systematic,  later  training  could  ultimately  focus  on  experiential  learning  (i.e.  many  case  studies) 
which  would  furnish  the  Staff  with  the  confidence  to  apply  the  IOPP  intuitively. 

3.3.1  Preliminary  validation 

The  content  of  the  IOPP  training  was  reviewed  by  the  Scientific  Authority  as  well  as  a  military 
SME  who  was  involved  in  the  creation  of  the  IOPP  and  knowledgeable  of  the  OPP.  This  is  the 
same  SME  referred  to  earlier  in  the  paper  and  is  one  of  the  authors  of  this  paper. 

3.2  Additional  materials 

In  addition  to  the  PowerPoint  training  courses  themselves,  additional  materials  were  provided  for 
participants  on  the  training  website.  These  materials  included  a  summary  of  instructions  to 
download  and  complete  the  training,  a  glossary  of  terms  used  in  both  the  OPP  and  IOPP  training 
that  could  be  downloaded  from  the  site  and  either  saved  or  printed  for  reference,  a  copy  of  the 
OPP  and  IOPP  models  for  participants  to  download  or  print  for  reference,  and  an  online 
participant  feedback  questionnaire.  Participants  were  requested  to  complete  the  questionnaire 
following  the  completion  of  both  the  OPP  and  IOPP  training. 

3.3  Pilot  testing 

In  the  first  phase  of  pilot  testing,  both  the  OPP  and  IOPP  training  courses  were  pilot  tested 
internally  to  allow  an  approximation  of  level  of  effort  (i.e.  amount  of  time)  to  complete  each 
course.  Two  internal  participants  (i.e.  employees  of  Humansy stems®),  with  no  familiarity  with 
the  OPP,  were  able  to  complete  each  of  the  OPP  and  IOPP  training  courses  in  approximately  two 
hours.  We  did  not  test  participant  retention  of  the  training  as  the  aim  of  this  pilot  testing  was  to 
identify  formatting  or  spelling  errors  as  well  as  predict  the  level  of  effort  to  complete  the 
training.  Further,  we  felt  that  non-military  people  would  not  retain  the  material  as  well  as 
military  people. 

The  second  phase  of  pilot  testing  involved  external  participants  (military  personnel  with 
knowledge  of  the  OPP)  accessing  the  website,  downloading  the  training  and  additional  materials 
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and  completing  the  training  modules.  The  intent  of  this  training  was  to  approximate  the  level  of 
effort  required  by  military  personnel  to  complete  the  training  courses  and  also  to  detect  any 
technical  difficulties  related  to  downloading  materials,  emailing  the  administrator  and  collecting 
web  statistics.  This  pilot  testing  confirmed  that  participants  were  able  to  email  the  administrator 
and  web  statistics  could  in  fact  be  collected.  The  testing  also  identified  certain  technical 
difficulties  related  to  downloading  the  training  courses  which  were  then  investigated  and  resolved 
by  the  HSI®  project  team. 

3.4  Results  of  training  with  participants  of  experimental  evaluation 

All  12  of  the  participants  of  the  experimental  evaluation  (refer  to  Section  4  of  the  report)  were 
able  to  successfully  log  onto  the  website  and  download  the  training  courses.  Although  web 
statistics,  such  as  the  time  of  login  and  logoff,  were  successfully  captured,  due  to  the  format  of 
the  data  collected,  it  was  impossible  to  tell  if  participants  were  logging  off  from  one  session  or 
logging  in  for  another  session.  Therefore  any  calculations  of  average  time  to  complete  training 
would  not  necessarily  be  valid  because  most  participants  logged  in  and  out  multiple  times. 
Furthermore,  participants  were  unable  to  submit  the  feedback  questionnaire.  These  faults  should 
be  addressed  in  future  experiments  involving  the  OPP  and  IOPP  online  training. 

3.4.1  Feedback  from  participants 

An  online  feedback  questionnaire  was  provided  as  part  of  the  online  training.  Participants  were 
requested  to  complete  the  feedback  questionnaire  after  completing  each  of  the  training  packages 
(i.e.  OPP  and  IOPP).  Participants  were  asked  to  rank  12  characteristics  of  the  training  on  a  five- 
point  scale  (1  =very  dissatisfied,  5= very  satisfied).  Unfortunately,  due  to  technical  difficulties, 
participants  were  unable  to  submit  the  questionnaire.  As  such,  participants  were  asked  to 
complete  a  hard  copy  of  the  feedback  questionnaire  during  the  experiment.  Only  7  feedback 
questionnaires  were  completed  and  the  feedback  provided  was  for  the  training  as  a  whole  rather 
than  for  the  OPP  and  IOPP  individually,  making  direct  comparisons  of  the  OPP  and  IOPP 
modules  impossible.  Figure  6  below  shows  the  average  rating  for  each  characteristic  and 
individual  participant  ratings  are  provided  in  Annex  B.  Results  suggest  that  participants  were 
satisfied  with  the  convenience  of  the  training,  relevance  of  the  training  to  their  work,  and  the 
information  imparted  in  the  training.  Even  though  participants  like  the  convenience  of  the  web- 
based  training,  during  the  AARs,  a  number  of  participants  stated  that  they  would  prefer  lecture- 
based  training  with  an  instructor  present. 

Participants  expressed  the  greatest  dissatisfaction  with  the  inability  to  have  questions  answered  in 
real  time  during  the  training  and  the  level  of  detail  of  the  examples  (e.g.  of  planning  products) 
used  in  the  training.  Participants  were  told  that  emails  received  between  0830h-1700h  Monday- 
Friday  would  be  responded  to  within  two  hours.  Outside  of  those  hours,  participant  emails 
would  be  responded  to  as  soon  as  possible.  It  appears  that  participants  would  have  preferred 
more  availability  by  the  administrator.  It  was  also  noted  that  some  participants  were  not  able  to 
email  the  administrator  due  to  technical  issues.  With  respect  to  the  use  of  examples,  comments 
made  during  the  After- Action  Reviews  (AARs)  conducted  during  the  experimental  evaluation 
suggest  that  participants  would  have  liked  more  concrete  examples,  especially  of  the  planning 
products  or  output.  In  fact,  one  team  suggested  the  use  of  JPEG  or  MPEG  files  showing  specific 
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planning  products  either  being  produced  or  completed.  Participants  also  suggested  the  use  of  one 
standard  example  for  both  training  processes  to  allow  easy  comparison. 


Participant  feedback  on  training 


in 


05 

c 


O  o 
© 

*  0 


"O 

CD 

CO 

cd 

_Q 


-Q  CD 
CD  -o 


CD 

O 

c 

© 

c 

CD 

> 

C= 

o 

O 


CD 


(/) 
(/) 
CD 

CD  | 
> 

O  o 

CD 


O  -O 
CD  CD 
-Q  -O 
■D  > 
CD  o 
CD  c 
LL  Q- 


O  E 

"O  CD 


'E  to 
id  0 

t  =5 

o  o- 

CL 

O-  0 
O  cd 


o-o  ° 

£  r  o 

E  CD  c 

5-  CL  CD 

£  F  > 

C  ■- 


^  o 

1 1 
>■§ 


Figure  6:  Mean  participant  ratings  of  online  training 

Other  participant  comments  that  were  raised  during  the  AARs  regarding  the  web-based  training 
include: 

•  Not  enough  questions  to  confirm  retention; 

•  Acronyms  not  explained  and  not  in  glossary; 

•  Graphics  written  over  text;  and 

•  OPP/IOPP  lessons  were  for  a  Brigade  staff  but  it  was  applied  at  a  battle  group  level. 

The  last  comment  suggests  that  the  training  may  not  have  been  relevant  for  the  context  of  the 
exercise  in  which  the  OPP  and  IOPP  were  applied  (i.e.  battle  group  HQ).  This  should  be  taken 
into  account  for  future  experiments. 

Given  the  available  time  and  resources,  however,  the  online  training  was  a  significant 
accomplishment.  Future  opportunities  to  develop  an  online  or  distance  learning  course  for  the 
IOPP  and  the  OPP  would  build  on  the  work  already  done;  it  would  not  adopt  a  significantly 
different  approach  or  significantly  different  content. 
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4.  Design  and  Method  of  the  Experimental 
Evaluation  of  the  Intuitive  Planning 
Process 


An  experimental  evaluation  of  the  IOPP  was  conducted  to  compare  the  performance  of  planning 
teams  using  the  OPP  and  the  IOPP. 

4.1  Method 

4.1.1  Participants 

A  total  of  12  participants  from  the  reserve  CIMIC  (Civil  Military  Cooperation)  unit  of  Land 
Force  Central  Area  (LFCA),  were  randomly  assigned  to  two  teams  of  six,  each  representing  a 
planning  team  at  the  Task  Force  (i.e.  Battle  Group)  level.  Specific  roles  of  the  staff  within  each 
team  included: 

1 .  The  Commander; 

2.  10  Officer; 

3.  Ops  Officer; 

4.  ISTAR; 

5.  Fires  (gunner);  and 

6.  Logistics. 

All  participants  completed  both  OPP  and  IOPP  online  training  prior  to  the  experiment.  As  well, 
all  participants  had  completed  formal  training  in  the  OPP  at  either  the  Canadian  Forces  College 
(CFC)  or  the  Canadian  Land  Force  Command  and  Staff  College  (CLFCSC)  in  Kingston. 
Immediately  prior  to  each  experimental  session,  all  participants  received  one  hour  of  refresher 
training  on  the  planning  process  they  would  be  using  for  that  session  (i.e.  IOPP  or  OPP). 

Prior  to  the  experimental  sessions,  participants  were  asked  to  complete  a  background 
questionnaire  detailing  their  participation  in  planning  exercises  or  operations  as  well  as  the 
amount  of  training  they  have  received  in  the  OPP.  Although  it  was  requested  that  all  participants 
have  a  basic  familiarity  with  the  OPP,  results  of  the  background  questionnaire  revealed  that  four 
participants  had  never  received  any  formal  training  in  the  OPP.  Three  participants  reported  that 
they  have  received  a  moderate  amount  of  training  in  the  OPP,  while  only  one  reported  having 
considerable  training  experience  with  the  OPP.  The  remaining  three  participants  reported  that 
they  have  attended  one  or  two  lectures  on  the  OPP,  although  the  location  (e.g.  CFC)  nor  context 
(e.g.  part  of  advanced  military  studies  course)  of  the  lectures  was  reported.  In  terms  of 
operational  experience  using  the  OPP,  five  participants  reported  having  no  operational 
experience,  six  participants  reported  some  experience  and  one  reported  a  moderate  amount  of 
experience. 
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4.1.2  Observers 

A  total  of  four  observers  from  the  HSI®  team,  along  with  two  research  assistants  (RAs)  from 
DRDC  Toronto,  were  involved  in  the  experimental  evaluation.  One  observer  and  one  RA  were 
available  for  each  planning  team  on  each  day  of  the  training  (two  days  total  with  the  two  planning 
teams  working  concurrently).  The  HSI  observers  stayed  with  the  team  throughout  the  two  days, 
while  the  DRDC  RAs  stayed  with  the  process  being  applied  (and  thus  observed  two  different 
teams). 

Two  military  SMEs  acted  as  expert  evaluators  throughout  the  experimental  exercises.  One 
evaluator  was  present  for  each  experimental  session  (i.e.  one  team  with  either  the  OPP  or  IOPP). 

A  scenario  description  as  well  as  well  as  orders  from  higher  command  were  provided  to  all 
observers  including  RAs  and  expert  evaluators.  This  permitted  observers  to  know  what  key 
factors,  activities,  or  behaviours  are  expected  so  as  to  make  their  rating  task  easier.  In  addition, 
all  observers  received  a  training  session  on  Measures  of  Performance  (MOPs)  to  capture,  how  to 
identify  MOPs,  etc. 

4.1.3  Design 

One  group  of  6  participants  was  named  ‘yellow’  and  the  other  group  named  ‘blue’.  A  mixed 
between  and  within-groups  design  was  adopted.  To  control  for  learning  and/or  order  effects,  the 
Blue  team  applied  the  OPP  on  the  first  day  of  the  experiment  and  the  IOPP  on  the  second  day, 
whereas  the  Yellow  team  applied  the  IOPP  on  the  first  day  and  the  OPP  on  the  second  day.  Both 
groups  were  presented  with  the  same  scenario  (a  defensive  operation)  on  the  first  day,  and  the 
same  scenario  (an  offensive  operation)  on  the  second  day.  Each  group  was  assessed  on  planning 
effectiveness,  planning  efficiency,  and  overall  performance.  Between  the  groups,  the  quality  of 
the  planning  products,  as  well  as  time-based  measures  of  efficiency,  were  used  to  compare  the 
relative  effectiveness  of  the  OPP  and  the  IOPP.  Specific  measures  of  performance  (MOPs)  are 
outlined  in  section  4. 1 .7  of  this  document. 

4.1.4  Equipment  and  materials 

The  simulated  planning  experiment  took  place  in  two  separate  training  rooms  at  CFC  in  Toronto. 
Equipment  provided  for  each  team  included  2  laptops  containing  software  for  word  processing, 
making  presentations,  creating  spreadsheets  and  visualizing  data,  1  whiteboard,  1  flip  chart  with 
paper,  talc  (for  creating  overlays),  pens  and  pencils,  etc. 

The  planning  teams  used  the  laptops,  whiteboards,  flipcharts  and  talc  for  the  majority  of  their 
planning  activities.  Products  such  as  the  Commander’s  Planning  Guidance,  CONOPS  and 
Orders  were  created  using  available  word  processing  software  or  handwritten  on  paper.  Planning 
tools  (e.g.  COA  comparison  matrix)  were  produced  using  commercially-available  software. 

Presentation  of  the  orders  at  the  completion  of  each  planning  session  was  videotaped  and  audio 
recorded  for  the  purpose  of  post-exercise  analysis  only.  Therefore  two  sets  of  video  and  audio 
recording  equipment  were  required  for  the  duration  of  the  experiment  (one  for  each  team). 

At  the  start  of  each  experimental  task  (i.e.  planning  session),  participants  were  provided  with  the 
following  materials  and  information: 
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•  A  general  narrative  of  the  exercise  describing  the  current  situation  as  well  as  opposing 
and  own  force  activities  up  to  the  point  of  the  beginning  of  the  scenario.  It  also  defined 
the  current  scenario  time; 

•  The  start  states  for  own  forces; 

•  ORBATs  (Order  of  Battle)  for  own  and  enemy  forces; 

•  Area  map  (1:50,000)  showing  the  Area  of  Operations  (AOO)  and  the  geography  and 
terrain; 

•  Decision  support  template; 

•  Synchronization  matrix; 

•  An  OP  O  (Operations  Order)  from  highest  command; 

•  A  Frago  O  (Fragmentary  Order)  from  higher  command  that  outlines,  among  others,  the 
situation,  mission  and  execution; 

•  Intelligence  Summary  (INTSUM); 

•  Task  Organization  Matrix; 

•  ISTAR  Matrix; 

•  Attack  Guidance  Matrix; 

•  Groupings  and  Taskings  Matrix; 

•  Overlay  (photocopied  onto  8.5  x  11  paper)  showing  blue  forces,  and, 

•  A  poster  showing  the  OPP  or  the  IOPP  (depending  on  the  experimental  condition). 

During  both  the  OPP  and  IOPP  planning  sessions,  the  participants  were  able  to  use  their  own 
resources  required  to  perform  the  planning  task  (e.g.  Battle  Staff  Smartbook). 

4.1.5  Experimental  Task 

The  simulated  planning  experiment  took  place  on  February  18  and  19  at  CFC  in  Toronto. 
Participants  played  roles  in  one  of  two  simulated  planning  staffs  and  created  a  plan,  using  both 
the  OPP  and  IOPP  for  two  scenarios.  The  two  planning  teams  were  located  in  separate  rooms 
(see  Figure  7). 
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Figure  7:  Experimental  facilities  for  Yellow  team 

Each  team  was  presented  with  a  scenario  and  given  a  mission  and  the  Commander’s  planning 
guidance  and  then  asked  to  create  a  plan  for  lower  formation(s).  Initial  mission  tasking  (i.e. 
HICOM)  and  Commander’s  guidance  was  provided  by  the  LCol  (Ret’d)  on  the  HSI®  team. 

The  manner  in  which  each  team  followed  and  applied  the  planning  process  was  observed  and 
documented  by  HSI®  team  observers,  RAs  and  the  two  expert  evaluators.  In  general,  the 
observers  documented  timings,  steps  in  the  planning  process  that  were  followed  (and  steps  that 
weren’t  followed),  errors  and  any  other  relevant  observations.  A  complete  inventory  of  MOPs 
collected  by  observers  as  well  as  participants  is  provided  in  section  4.1.7. 

Parallel  experimental  sessions  were  conducted  in  order  to  minimise  the  impact  on  project 
resources.  That  is,  the  two  planning  teams  performed  the  experimental  task  concurrently, 
although  in  separate  rooms. 

The  experimental  session  took  place  over  two  days  (0800h  -  1600h)  and  included  refresher 
training  on  both  the  OPP  and  IOPP.  A  general  experimental  briefing  was  held  on  an  evening  the 
week  prior  to  the  experiment.  At  this  point  a  general  description  of  the  scenario  was  provided  as 
well  as  an  overview  of  the  logistics  of  the  experiment  (expected  hours,  etc.).  Each  team  was 
asked  to  run  through  at  least  one  planning  cycle  for  both  the  OPP  and  IOPP. 
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4. 1.5.1  Planning  products 

The  planning  products  created  by  the  participants  during  each  experimental  session  differed  for 
the  OPP  and  IOPP,  as  certain  OPP  products  are  not  relevant  when  using  the  IOPP  for  planning. 
Table  2  shows  the  outputs  that  the  planning  teams  were  asked  to  produce  for  both  OPP  and  IOPP 
experimental  sessions. 


Table  2:  Planning  Products  for  the  OPP  and  IOPP 


OPP 

IOPP 

Initial  Intent 

Initial  Intent 

Wng  0  to  subordinate  formations 

Summary  of  Mission  Analysis  (replaces 

Mission  Analysis  Brief) 

Mission  Analysis  Brief  including  mission 
analysis  and  key  deductions 

Most  likely  enemy  COA 

Commander’s  Planning  Guidance 

Skeleton  COA 

Information  Brief  (with  three  friendly  and 
three  enemy  COAs,  including  most 
dangerous  and  most  likely) 

Criteria  for  success 

Decision  Brief 

Refined  COA 

CONOPS 

Final,  validated  COA 

Plan/Orders 

CONOPS 

Final  Plan 

Branch  and  sequel  plans  (if  applicable) 

Orders 

4.1.6  Scenarios 

Two  separate  paper-based  scenarios  were  used  to  avoid  potential  learning  effects.  The  use  of  the 
scenarios  was  counterbalanced  such  that  planning  team  yellow  used  the  OPP  with  scenario  A  then 
the  IOPP  with  scenario  B,  whereas  team  blue  used  the  IOPP  with  scenario  A  then  the  OPP  with 
scenario  B.  The  scenarios  were  adapted  from  existing  CF  scenarios  that  are  currently  used  for 
training  at  CLFCSC  in  Kingston.  As  such,  they  were  appropriate  for  the  Task  Force  (i.e.  Battle 
Group)  level.  One  scenario  entailed  a  defensive  operation  whereas  the  other  involved  an 
offensive  operation. 

The  scenarios  were  designed  to  be  of  similar  complexity  such  that  identical  MOPs  were 
applicable  to  both  experimental  conditions  and  both  scenarios.  In  order  for  us  to  be  able  to  assess 
the  perceived  similarity  of  the  scenarios  in  terms  of  complexity,  a  measure  was  used  that  allowed 
the  participants  to  rate  the  level  of  complexity  of  each  scenario  (i.e.  manipulation  check;  see 
Section  4.1.7). 

4.1.7  Data  Collected 

During  the  experimental  sessions,  the  observers  and  SMEs  documented  the  actions  of  their 
respective  planning  teams.  Within  each  team,  the  three  observers  (including  HSI®  and  DRDC 
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observers)  divided  responsibility  for  observing  participants  such  that  each  observer  was 
responsible  for  following  two  team  members  and  documenting  their  activities.  In  general,  the 
participants  observed  were  paired  according  to  their  role  role  and  with  whom  he  was  most  likely 
to  be  collaborating.  For  example,  one  observer  was  responsible  for  the  OpsO  and  AOpsO  given 
that  they  worked  most  closely  together.  Likewise,  the  Comd  and  10,  and  FOO  and  LogO  were 
paired  for  observation  purposes.  For  the  duration  of  each  experimental  session,  observers 
documented  the  time  (clock  or  according  to  a  countdown  timer),  activity  and  appropriate  OPP  or 
IOPP  step  performed.  The  majority  of  the  mapping  of  the  team  activity  to  appropriate  OPP  or 
IOPP  step  was  done  post  hoc  (although  some  observers  did  this  in  real-time).  The  two  SMEs 
were  also  asked  to  document  timings  and  observed  activities  for  the  duration  of  the  experiment. 
However,  as  there  was  just  one  SME  per  team,  their  observations  are  with  reference  to  the  entire 
planning  team  rather  than  two  individual  members  of  the  team.  The  observations  documented  by 
the  observers  and  SMEs  are  provided  in  Annex  C. 

One  limitation  of  this  observation  method  is  the  fact  that  all  OPP  and  IOPP  steps  are  not 
necessarily  explicit  and  therefore  it  may  be  difficult  for  observers  to  identify  exactly  which  step 
is  being  performed.  There  are  some  steps  that  are  certainly  more  observable  than  others.  For 
example,  formal  briefings  such  as  the  Mission  Analysis  Brief  (Step  2.6  in  OPP  and  2.7  in  IOPP) 
or  Decision  Brief  (Step  4.2  in  OPP)  are  easily  observable.  However,  functions  such  as  the 
Commander  deciding  if  he  has  sufficient  information  to  initiate  COA  Development  (Step  3.1  in 
IOPP),  or  the  decision  as  to  whether  the  staff  is  performing  crisis  action  planning  (Step  1.8  in 
OPP)  may  not  be  as  readily  observable. 

4. 1. 7. 1  Measures  of  Performance 

Measures  of  performance  enabled  a  direct  comparison  of  the  OPP  and  the  IOPP  across  a  number 
of  dimensions.  Given  the  design  of  this  experiment,  different  MOPs  were  collected  by  different 
individuals  in  order  to  evaluate  the  planning  process.  Specifically,  MOPs  were  divided  into 
measures  that  were  generated  by  expert  (military  SME)  evaluators  who  were  observing  the 
experiment,  measures  generated  by  the  participants  themselves,  and  measures  generated  by  the 
HSI®  team  of  observers  and  DRDC  research  assistants  (referred  to  collectively  as  observers). 
MOPs  addressed  both  the  planning  process  and  planning  products. 
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Table  3  outlines  a  number  of  MOPs  that  SME  evaluators  completed  during  both  experimental 
conditions  (i.e.  both  planning  processes).  MOPs  for  SME  evaluators  included  subjective 
measures  related  to  the  quality  and  efficiency  of  the  planning  process  and  the  quality  of  the 
planning  products  (i.e.  the  Plan  or  Orders).  Rating  scales  used  by  the  SME  evaluators  were  in 
the  form  of  a  modified  Behaviourally- Anchored  Rating  Scale  (BARS).  Each  rating  scale  listed  a 
number  of  behaviours  or  characteristics  to  look  for  in  rating  the  factor  (e.g.  quality  of  plan)  but 
did  not  provide  specific  behavioural  descriptors  for  each  rating  value  (i.e.  what  behaviour  is 
associated  with  “poor”).  The  following  section  includes  a  discussion  of  BARS  as  a  rating  scale 
methodology.  The  BARS  used  by  the  SME  evaluators  for  the  purpose  of  this  experiment  are 
shown  in  Annex  D. 
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Table  3:  MOPs  used  by  expert  SME  evaluators 


Dimension 

Potential  objective  measures 

Potential  subjective  measures 

Planning  process 

Number  of  steps  in  planning  process 
missed,  repeated,  performed  out  of 
order  or  concurrently 

Rating  of  perceived  quality  of  Mission 
Analysis  process1 

Rating  of  perceived  quality  of 
Commander’s  guidance 

Rating  of  perceived  quality  of  Staff 
Analysis  process 

Rating  of  perceived  quality  of  COA 
Evaluation/Validation  process 

Rating  of  overall  efficiency  of  team 
when  applying  planning  process 

Planning  product(s) 

N/A 

Rating  of  quality  of  selected  COA 

Rating  of  quality  of  CONOP 

Rating  of  overall  quality  of  Plan 

Rating  of  overall  quality  of  Order 

Table  4  outlines  a  number  of  MOPs  that  were  collected  by  HSI®  and  DRDC  observers  while 
observing  both  experimental  conditions  (i.e.  both  planning  processes).  This  list  includes  only 
objective  measures  of  the  planning  process  itself,  such  as  time  to  complete  plan,  number  of 
communications,  etc. 


Table  4:  MOPs  used  by  HSI®  team  observers  and  RAs 


Dimension 

Potential  objective  measures 

Potential  subjective  measures 

Planning  process 

Time  to  select  COA 

Time  to  complete  CONOPS 

Time  to  complete  Plan 

Time  to  complete  Orders 

Requests  for  additional  information  or  clarification 

Number  of  steps  in  planning  process  missed, 
repeated,  performed  out  of  order  or  concurrently 

Total  errors  made  in  planning  process 

Number  of  team  briefings  observed  -  both  formal 
and  no-formal  (Note  number  of  participants) 

Number  of  options  presented  to  commander 

Number  of  non-mandated  steps  added  to  the 
process 

N/A 

Table  5  outlines  the  MOPs  that  were  completed  by  the  participants,  including: 


1  This  would  also  be  an  indicator  and  a  potential  measure  of  situation  awareness 
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•  Subjective  and  objective  measures  of  the  planning  process  (e.g.  usability,  user 
trust/confidence,  reliability,  etc.); 

•  Subjective  measures  of  the  planning  products  (e.g.  user  trust/confidence,  reliability, 
overall  quality,  etc.);  and 

•  Manipulation  checks  of  scenario  complexity  and  realism  of  time  pressure. 

Participant  MOPs,  other  than  workload  measures,  were  collected  at  the  end  of  the  day  for  each 
experimental  condition  (i.e.  OPP  and  IOPP).  Rating  scales  for  participants  were  created  using  a 
Likert  5-point  scale  and  are  shown  in  Annex  E.  Rather  than  using  standard  anchor  terms, 
anchors  were  selected  according  to  the  dimension  being  rated.  BARS  were  not  used  for 
participant  MOPs. 

Workload  measures,  using  the  NASA  TLX  (Task  Load  Index)  Questionnaire,  were  completed 
by  the  participants  twice  for  each  experimental  condition;  halfway  through  the  day  and  at  the  end 
of  the  day. 


Table  5:  MOPs  used  by  Participants 


Dimension 

Potential  objective  measures 

Potential  subjective  measures 

Planning  Process 

Usability 

N/A 

Rating  of  ease  of  use  of  planning  process 
Rating  of  learnability  of  planning  process 
Rating  of  efficiency  of  planning  process 

Rating  of  effectiveness  of  planning 
process 

Rating  of  overall  user  satisfaction  with 
planning  process 

User  trust/confidence  and  reliability 

N/A 

Rating  of  user  trust/confidence  in  planning 
process 

Rating  of  reliability  of  planning  process  to 
produce  a  good  plan 

Accuracy/Errors 

N/A 

Rating  of  potential  for  planning  process  to 
lead  to  errors  in  overall  plan 

Rating  of  potential  criticality  of  errors  in 
overall  plan  that  may  be  associated  with 
planning  process 

Operational  effectiveness 

N/A 

Rating  of  suitability  of  planning  process  in 
a  variety  of  operational  contexts 

Rating  of  degree  to  which  the  team 
followed  the  process 

Workload 

N/A 

Rating  of  complexity  of  planning  process 
Real  time  subjective  ratings  of  workload 
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Planning  Product(s) 

User  trust/confidence  and  reliability 

N/A 

Rating  of  user  trust/confidence  in  the 
overall  Plan  to  complete  the  mission 

Rating  of  reliability  the  overall  Plan  to 
complete  the  mission 

Accuracy/Errors 

N/A 

Rating  of  accuracy  of  the  overall  Plan  in 
completing  the  mission 

Overall  quality 

N/A 

Rating  of  overall  quality  of  Plan 

Manipulation  Checks 

N/A 

Rating  of  realism  of  scenario 

Rating  of  realism  of  time  available  for 
planning 

4. 1.7. 2  Rating  Scales 

Meister  (1985)  notes  that  rating  scales  may  be  used  to: 

•  Evaluate  how  well  a  task  is  being  performed  (e.g.  following  a  plan  precisely) 

•  To  evaluate  some  quality  of  performance  (e.g.  leadership  of  the  OC  during  mission 
planning); 

•  Quantify  the  adequacy  of  some  feature  of  a  system  (e.g.  the  detail  of  navigational  plan); 

•  Evaluate  the  effect  of  some  condition  (e.g.  critiquing  aid  versus  existing  practice);  and/or 

•  Evaluate  the  output  of  performance  (e.g.  the  choice  of  a  tactical  strategy). 

While  there  are  a  number  of  approaches  to  the  construction  of  ratings  scales,  BARS  have 
emerged  as  a  preferred  technique  because  the  behavioural  descriptors  assigned  to  each  item’s 
rating  scale  are  thought  to  provide  the  necessary  anchors  to  enhance  the  precision  of  the  rating 
and  to  standardise  across  observers. 

BARS  have  also  been  shown  to  have  good  construct  validity,  and,  when  used  to  assess  planning 
performance,  have  demonstrated  a  high  correlation  with  assessments  of  actual  operational 
performance  in  air  combat  missions  (Spiker,  Nullmeyer  and  Tourville,  2001).  An  example  BAR 
scale  is  shown  in  Figure  8. 

BARS  has  definite  advantages  over  simply  asking  judges  to  “rate  on  a  scale  of  1-5  the  quality  of 
mission  briefings”.  First,  the  scale  defines  the  behaviour  category  of  interest,  in  this  case 
“Quality  of  the  Mission  Analysis  Process”.  Second,  it  summarises  the  range  of  behaviours  of 
interest  relating  to  the  quality  of  the  mission  analysis  process  i.e.  the  number  of  factors 
considered,  the  relevance  of  the  factor  considered  and  the  amount  of  insight  brought  to  the 
analysis.  Third,  it  provides  descriptors  of  the  performance  standard  that  merit  the  different 
rating  values.  Fourth,  it  directs  the  rater  to  specific  key  behaviours.  Because  of  these 
advantages,  BARS  were  used  for  SME  evaluators  MOPs.. 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very  Good 

5.Exceptional 

Quality  of  the  Mission  Analysis  Process 

Many  relevant  factors 
not  considered 

Several  factors  not 
considered,  including 
some  relevant  factors 

The  most  obvious 
factors  were 
considered  but  no 
attempt  was  made  to 
think  ‘out  of  the  box’ 
nor  were  interaction 
effects  between  factors 
considered 

All  obvious  and  some 
peripheral  but 
significant  factors 
considered;  interaction 
effects  between  factors 
considered 

All  potentially 
significant  factors  and 
their  interactions 
considered;  high 
degree  of  insight 
brought  to  analysis 

Look  for: 

Many  different  factors  noted  during  discussion 

Consideration  of  how  factors  might  influence  each  other 

Inferences  made  on  the  basis  of  known  facts 

Appropriateness  of  factors  considered 

Observations: 

Figure  8:  Example  BARS 


4. 1.7. 3  Workload  Measure 

The  NASA  TLX,  which  is  a  retrospective  measure  of  mental  workload  (Hart  &  Staveland, 

1988),  was  administered  at  the  mid-point  and  end  (immediately  following  each  experimental 
session)  of  each  day.  This  measure  was  used  to  compare  workload  across  a  number  of 
dimensions  including  the  OPP  and  IOPP.  A  retrospective  assessment  was  preferable  to  a  real¬ 
time  measure  for  this  experiment  given  that  the  the  work  (i.e.  planning)  was  mostly  self-paced. 
The  NASA  TLX  collects  ratings  for  six  dimensions  (mental  demand,  physical  demand,  temporal 
demand,  performance,  effort,  frustration)  along  a  continuous  scale.  Each  dimension  is  compared 
with  all  other  dimensions  in  a  pairwise  manner  to  generate  a  weighting  for  each  dimension.  This 
permits  calculation  of  a  weighted  composite  workload  score. 

4.1.8  After-Action  Review 

Upon  completion  of  each  experimental,  an  AAR  was  conducted  with  all  participants  in  each 
team,  SME  evaluators,  and  observers.  The  purpose  of  the  AAR  was  to  provide  an  opportunity  to 
discuss  the  experimental  sessions  as  well  as  elicit  feedback  from  the  participants  on  the  IOPP 
compared  to  the  OPP  and  other  relevant  topics. 


4.1.9  Schedule 

The  experimental  evaluation  took  place  over  two  days,  with  different  planning  teams  performing 
the  experimental  task  concurrently.  As  illustrated  in  Table  6,  the  morning  of  each  day  consisted 
of  refresher  training  on  both  IOPP  and  OPP  along  with  provision  of  the  scenario  to  be  used  for 
the  experimental  session.  For  the  remainder  of  the  first  day,  Team  Blue  performed  the 
experimental  task  using  the  OPP,  while  Team  Yellow  did  the  same  using  the  IOPP.  On  day  two, 
Team  Blue  performed  the  experimental  task  using  the  IOPP  and  with  a  new  scenario  while  Team 
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Yellow  used  the  OPP  with  a  new  scenario.  Working  hours  for  each  day  of  the  experiment  was 
0800h  -  1600h  with  approximately  a  half  hour  for  lunch. 


Table  6:  Schedule  for  the  experimental  evaluation 


Time 

Day  1 

Day  2 

Team  Blue 

Team  Yellow 

Team  Blue 

Team  Yellow 

0800-0830h 

Welcome  /  OPP 
refresher 

Welcome  /  IOPP 
refresher 

IOPP  refresher 

OPP  refresher 

0830-0900h 

Scenario  briefing 

Scenario  briefing 

Scenario  briefing 

Scenario  briefing 

0930h-1530h 

Experimental  task 
using  OPP  and 
scenario  A 

Experimental  task 
using  IOPP  and 
scenario  A 

Experimental  task 
using  IOPP  and 
scenario  B 

Experimental 
task  using  OPP 
and  scenario  B 

1 530-1 600h 

AAR 

AAR 

AAR 

AAR 

4.2  Results 

4.2.1  Data  Analysis 

Observers  and  SMEs  documented  actions  observed,  the  time  of  action  (clock  or  according  to  the 
countdown  timer),  and  the  corresponding  OPP  or  IOPP  step.  The  majority  of  the  mapping  of  the 
team  activity  to  appropriate  OPP  or  IOPP  step  was  done  post  hoc.  Mapping  was  done  by 
individual  observers  and  SMEs  but  was  verified  by  an  internal  SME. 

The  data  collected  by  each  observer  were  aggregated  and  expressed  according  to  a  standardized 
timeline.  The  result  was  one  data  set  including  time,  actions  observed  and  OPP/IOPP  step  for 
each  experimental  session  (i.e.  each  team  and  each  day).  That  is,  four  data  sets  were  produced: 
Team  Yellow  OPP,  Team  Yellow  IOPP,  Team  Blue  OPP,  Team  Blue  IOPP.  The  following 
observer  MOPs,  outlined  in  Table  4,  were  calculated  from  these  four  data  sets: 

•  Time  to  select  CO  A; 

•  Time  to  complete  CONOPS; 

•  Time  to  complete  Plan; 

•  Time  to  complete  Orders; 

•  Requests  for  additional  information  or  clarification; 

•  Number  of  steps  in  planning  process  missed,  repeated,  performed  out  of  order  or 
concurrently  ; 

•  Total  errors  made  in  planning  process; 

•  Number  of  team  briefings  observed  -  both  formal  and  no-formal  (Note  number  of 
participants); 

•  Number  of  options  presented  to  commander;  and 

•  Number  of  non-mandated  steps  added  to  the  process 

The  MOPs  generated  by  the  SME  evaluators  (outlined  in  Table  3)  were  collected  and  analyzed 
individually  and  then  combined  for  further  analysis. 
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Participant  ratings  of  the  planning  process  and  products  (outlined  in  Table  5),  as  well  as 
workload  ratings  collected  by  means  of  the  NASA  TLX,  were  combined  for  further  analysis. 

Data  collected  from  the  SME  evaluators,  observers  and  participants  were  compared  across  a 
number  of  dimensions  to  identify  differences,  especially  differences  between  the  OPP  and  IOPP. 
Data  from  the  two  experimental  teams  was  also  considered  for  consistency.  Given  the  sample 
size,  descriptive,  rather  than  statistical  analyses  were  performed  on  the  data. 

In  addition,  all  notes,  overlays,  diagrams,  etc.  created  by  the  participants  during  the  planning 
exercise,  along  with  video  and  audio  recordings  of  presentation  of  the  orders,  were  reviewed  and 
used  to  supplement  and  embellish  the  results  of  the  MOPs  and  overall  discussion  points. 

4.2.2  Realism  of  scenario  and  time  available 

As  a  manipulation  check,  participants  were  asked  to  rate  the  realism  of  the  scenarios  as  well  as 
that  of  the  time  available.  Individual  participant  ratings  for  both  scenarios  are  provided  in  Annex 
H.  Figure  9  below  shows  the  average  rating  of  realism  of  scenario  and  time  available  for 
scenario  A  (defensive)  and  scenario  B  (offensive).  Overall,  participants  found  the  offensive 
scenario  more  realistic  than  the  defensive  scenario  and  that  they  had  a  more  realistic  amount  of 
time  available  for  the  offensive  than  defensive  scenario.  This  suggests  that  the  scenarios  may  not 
have  been  equivalent  in  terms  of  complexity  or  other  characteristics.  Given  that  the  teams  used  a 
different  scenario  for  the  OPP  and  IOPP,  it  is  possible  that  differences  in  the  scenarios  may  have 
contributed  to  some  of  the  differences  in  performance  or  quality  of  planning  process  or  products. 
Ratings  of  both  scenarios,  however,  were  above  the  mid-point  of  the  scale,  indicating  that 
participants  found  both  scenarios  to  have  a  reasonable  degree  of  realism. 


Participant  rating  of  manipulation  checks 
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Figure  9:  Ratings  of  realism  of  scenario  and  time  available 
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4.2.3  Overview  of  experimental  sessions  and  planning  teams 

Each  team  represented  a  battlegroup  staff.  The  composition  of  the  planning  staff  for  yellow  team 
was  a  Commander  (Comd),  an  Operations  Office  (Ops  O),  an  Assistant  Operations  Officer 
(AOps  O),  a  Fire  Officer  (FOO)  or  Fire  Support  Coordinator,  Intelligence  Officer  (Int  O)  and  an 
Engineer.  The  composition  of  the  planning  staff  for  the  blue  team  was  a  Comd,  Ops  O,  AOps 
O,  FOO,  Int  O  and  Logistics  Officer  (Log  0).Both  teams  planned  for  a  defensive  scenario  on 
day  one.  On  this  day,  the  yellow  team  conducted  the  planning  exercise  using  the  IOPP  whereas 
the  blue  team  used  the  OPP.  On  day  two,  both  teams  planned  for  an  offensive  scenario,  with  the 
Yellow  team  using  the  OPP  and  the  Blue  team  using  the  IOPP.  The  yellow  team  had  a  total  of  6 
participants  for  the  first  day  then  5  participants  for  the  second  day  as  one  participant  was  unable 
to  attend.  The  blue  team  had  a  total  of  6  participants  on  both  days 

4.2.4  Overview  of  steps  performed 

The  OPP  and  IOPP  steps  performed  by  each  team  in  each  condition  were  determined  by 
combining  all  observer  data.  The  total  number  of  possible  steps  was  calculated  from  the  OPP 
and  IOPP  swimlane  diagrams  that  were  provided  to  the  teams  as  reference.  As  seen  in  Annex  F, 
the  swimlane  diagrams  show  tasks  performed  by  a  specific  individual  or  team  (i.e.  Higher 
Command,  Commander,  Planning  Staff,  Subordinate  Formations).  Each  function  or  decision 
point  in  the  OPP  and  IOPP  swimlane  diagrams  was  assigned  a  number  to  be  used  coding 
purposes  and  subsequent  calculation  of  the  sequence  of  steps  performed  (see  Annex  F). 

4.2.4  Total  number  of  OPP  and  IOPP  steps  performed 

The  total  number  of  OPP  and  IOPP  steps  performed  was  calculated  for  each  team  and  each 
experimental  condition.  These  data  were  then  averaged  across  teams  and  normalized  according 
to  the  total  number  of  possible  steps  for  the  OPP  and  IOPP  (see  Table  7).  For  the  purpose  of  this 
experiment,  teams  were  asked  to  terminate  the  OPP  at  the  end  of  Plan  Development  and  the 
IOPP  once  the  Orders  had  been  issued.  As  such,  all  steps  of  Plan  Review  (Step  6  of  the  IOPP) 
as  well  as  the  last  two  steps  of  the  IOPP  (i.e.  5.6  and  5.7)  were  not  counted  in  the  total  possible 
steps,  leaving  a  total  of  50  possible  OPP  steps  and  37  possible  IOPP  steps.  Overall,  the  teams 
performed  72%  of  OPP  steps  and  84%  of  IOPP  steps.2 


2  Calculated  as  a  percentage  of  total  steps  in  each  process,  the  teams  performed  64%  of  OPP  steps  and  79% 
of  IOPP  steps. 


Page  32 


Maple  Skies  Exercise  Management  Process 


Human  systems®  Incorporated 


HUMAHSYSTEMS 

lMe«rpnrMt*4 


Table  7:  Total  number  of  OPP  and  IOPP  steps  performed 


OPP 

IOPP 

Number 

Total  number 

Number 

Total  number 

of  steps 

of  possible 

of  steps 

of  possible 

followed 

steps 

followed 

steps 

Step  1 

6 

9 

6 

Step  2 

10 

10 

9 

Step  3 

11 

14 

4 

Step  4 

3 

4 

9 

Step  5 

6 

13 

3 

Step  6 

0 

64 

n/a 

n 

TOTAL 

36 

56 

31 

4.2.5  Time  to  complete  planning  process  and  planning  products 

The  overall  time  to  complete  the  planning  exercise  was  calculated  from  the  point  at  which  the 
team  actively  started  planning  following  the  scenario  briefing  until  the  Commander  had 
completed  his  presentation  of  the  final  orders.  The  total  time  therefore  included  the  lunch  break, 
which  was  relatively  consistent  across  teams.  The  times  to  complete  the  planning  exercise  by 
each  team  (Blue  or  Yellow)  for  each  day  (1  or  2)  are  shown  in  Figure  10.  Team  Yellow  took 
less  time  to  perform  the  planning  exercise  using  the  OPP  than  the  IOPP.  On  the  other  hand, 
Team  Blue  took  less  time  to  complete  the  planning  exercise  using  the  IOPP  than  with  the  OPP. 
Each  team  took  less  time  to  complete  the  planning  exercise  on  the  second  day  which  is  consistent 
with  a  practice  effect.  However,  the  Blue  team,  using  the  IOPP,  derived  a  larger  benefit  on  the 
second  day. 


3  Due  to  the  fact  that  participants  were  asked  to  stop  after  issuing  orders,  5  rather  than  7  possible  steps  could 
have  been  performed 

4  Due  to  the  context  of  the  experiment,  Step  6  of  the  OPP  was  not  performed 
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Time  to  complete  planning  process 


Day  1  Day  2 


Figure  10:  Total  time  to  complete  planning  process 

If  the  data  are  collapsed  across  days  and  teams,  the  total  time  to  complete  the  planning  process 
using  the  OPP  was  6.42  hours  and  6.04  hours  when  using  the  IOPP. 

The  average  time  to  complete  the  COA,  CONOP  and  Plans/Orders  using  the  OPP  and  IOPP  are 
shown  in  Table  8  as  well  as  Figure  11.  Times  to  complete  planning  products  were  averaged 
across  teams  as  well  as  observers  (from  a  total  of  four  observers).  Raw  data  from  each  observer 
are  provided  in  Annex  F. 

Figure  12  shows  the  difference  in  times  to  complete  each  planning  product  from  day  1  to  day  2. 
We  were  not  able  to  calculate  a  time  difference  for  the  Blue  team  producing  the  CONOP  as  they 
did  not  explicitly  produce  a  CONOP  on  day  1 .  This  data  shows  that  the  Blue  team  experienced  a 
significant  reduction  in  time  to  select  COA  and  time  to  complete  Plans/Orders  from  day  1  (OPP) 
to  day  2(IOPP).  The  Yellow  team,  however,  had  only  a  slight  reduction  in  time  from  day  1 
(IOPP)  to  day  2  (OPP)  in  completing  the  CONOP  and  Plans/Order  and,  in  fact,  took  longer  to 
select  the  COA  on  day  2  compared  to  day  1.  Therefore,  the  Blue  team,  which  used  the  IOPP  on 
day  2,  showed  a  larger  reduction  in  completion  times  (with  the  exception  of  time  to  complete 
CONOP)  than  the  Yellow  team  which  used  the  OPP  on  day  2.  This  suggests  that  there  was 
likely  a  practice  effect,  especially  for  the  Blue  team. 

Table  8:  Time  to  complete  planning  products  for  OPP  and  IOPP  conditions 


IOPP 

OPP 

Time  to  select  COA 

2:25 

5:59 

Time  to  complete  CONOP 

4:02 

5:24 

Time  to  complete  Plan/Orders 

5:45 

6:27 
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Average  time  to  complete  COA,  CONOP  and 
Plan/orders  for  OPP  and  IOPP 


Time  to  select  Time  to  complete  Time  to  complete 
COA  CONOP  Plan/Oders 


□  OPP 
■  IOPP 


Figure  11:  Average  time  to  complete  planning  products  for  the  OPP  and  IOPP 

Difference  in  time  (Day  1-  Day  2)  to  complete  COA,  CONOP  and  Plan/Orders 
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-200 


Time  to : 


ilect  COA 


Time  to  complete  CONOP  Time  to  complete  Plan/Orders 


Measures  of  Performance 
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Figure  12:  Difference  in  time  to  complete  planning  products  from  Day  1  to  Day  2 
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4.2.6  Steps  in  planning  process  missed,  repeated,  performed  out  of  order  or 
concurrently 

Figure  13  and  Figure  14  below  show  all  of  the  steps  performed  in  the  OPP  and  IOPP, 
respectively.  The  shaded  areas  in  each  table  indicate  steps  that  were  performed  in  full  by  either 
of  the  teams,  whereas  non-shaded  areas  indicate  steps  that  were  skipped  or  abbreviated.  Some  of 
the  steps,  for  both  the  OPP  and  IOPP,  were  not  performed  due  to  the  lack  of  realism  with  the 
planning  exercise.  For  example,  there  were  no  supporting/subordinate  formations  so  the  teams 
could  not  pass  orders  to  these  formations.  Likewise,  a  synchronization  wargame  with  lower 
formations,  as  outlined  in  the  IOPP,  was  not  performed.  In  addition,  given  that  there  were  no 
players  for  the  role  of  higher  command,  approval  of  the  CONOP  from  higher  command  was  not 
feasible. 

As  previously  noted,  teams  were  asked  to  follow  the  OPP  until  the  end  of  Step  5  (Plan 
Development)  and,  as  a  result,  none  of  the  steps  in  Plan  Review  were  performed.  Some  of  the 
key  OPP  steps  that  the  teams  chose  to  skip  or  abbreviate  include  testing  the  viability  of  own 
CO  As  (Step  3.4),  the  development  of  branch  or  sequel  plans  (Step  5.6,  5.7)  and  plan  wargame 
(Step  5.8).  This  is  consistent  with  previous  research  (Bruyn,  Rehak,  Vokac  &  Lamoureux, 

2005)  and  is  not  surprising  in  the  context  of  this  experiment  given  the  imposed  time  constraints. 
With  respect  to  the  IOPP,  some  of  the  key  steps  that  were  skipped  or  abbreviated  include  a  check 
that  the  Commander  has  sufficient  information  to  initiate  CO  A  development  (Step  3.1), 
Commander  approval  of  COA  and  criteria  for  success  (Step  3.3)  and,  a  check  that  the  COA  is 
refined  enough  to  do  staff  estimates  (Step  4.1). 


1.0  Initiation 

2.0  Orientation 

3.3  Staff  Develop  Initial  Enemy 
and  Own  COAs 

4.2  Decision  Brief 

(No  -go  to  5.11) 

1.1  Mission  Receipt 

2.1  Staff  Review  Situation 

3.4  Staff  Test  Viability  of  Own 

COAs 

4.2.1  Staff  Prepare  and  present 
decision  brief 

5.10  Higher  Comd  decides  if  Plan 
is  approved 

1.1.1  Mission  Receipt  from  Higher 
Cmd 

2.2  Staff  Review  Higher  Level 
Information 

3.5  Information  Brief 

4.2.2  Comd  Attends  decision  brief 

(Yes  -  go  to  5. 1 1) 

1.1.2  Comd  Initiates  Mission 

Receipt 

2.3  Staff  Develop  own  Information 
based  on  Higher 

3.5.1  Comd  Attends  Information 

Brief 

4.3  Comd  selects  COA 

(No-  return  to  5.1) 

1.2  Issue/Receive  Initiating 

2.4  Staff  Develop  Mission 

3.5.2  Staff  Prepare  and  Present 

4.4  Staff  Develop  Concept  of 

5.11  Revise  Final  Plan 

Directive 

Statement 

Information  Brief 

Operations 

1.2.1  Comd  Issues  Initiating 

2.5  Staff  Prepare  Mission  Analysis 

3.6  Comd/COS  provides  further 

5.0  Plan  Development 

5.12  Staff  decides  if 

Directive 

Brief 

guidance 

issues/shortfalls  need 
to  be  identified  and 

1.2.2  Staff  Receives  Initiating 

Directive 

2.6  Mission  Analysis  Briefing 

3.7  Staff  Continues  Checks  and 
Analyses  of  Own 

5.1  Staff  Further  Refines  Comd's 
intent  and  concept 

(Yes  -  return  to  5.4) 

1.3  Comd  and  Staff  Activate 

2.6.1  Staff  delivers  Mission 

3.8  Comd  decides  if  any  COAs 

5.2  Comd  seeks  CONOP  approval 

(No  -  go  to  5. 13) 

Planning  Staff 

Analysis  Briefing 

need  to  be  refined 

from  higher  authority 

1.4  Staff  Gathers  Planning  Tools 

2.6.2  Comd  receives  Mission 
Analysis  Briefing 

(Yes  -  go  to  3.9) 

5.3  Higher  Comd  decides  if 

CONOP  approved 

5.13  Issue  Final  Plan 

1.5  Issue/Receive  Guidelines 

2.7  Additional  Guidance 

(No  -  go  to  3. 10) 

(Yes  -  go  to  5.4) 

6.0  Plan  Review 

1.5.1  Comd  issues  Guidelines  to 

2.7.1  Comd  provides  additional 

3.9  Refine  COAs  selected  by 

(No  -  return  to  5.1) 

6.1  Staff  conducts  progress  review 

Staff 

guidance 

Comd  (as  required) 

of  operation 

1.5.2  Staff  Receives  Guidelines 

2.7.2  Staff  receives  additional 

3.10  Comd  decides  if  any  new 

5.4  Staff  identifies  and  resolves 

6.2  Staff  conducts  periodic 

from  Comd 

guidance 

COAs  need  to  be 
developed 

issues/shortfalls 

OPLAN/CONPLAN  for 
review 

1.6  Comd  makes  Initial  Assessment 

2.8  Comd  develops  Comd's 

Planning  Guidance 

(Yes  -  go  to  3. 1 1) 

5.5  Staff  Prepares  Plan 

6.3  Staff  conducts  detailed 

exercise/wargaming 

1.7  Initial  Comd's  Guidance 

2.9  Staff  Finalize  Mission 

Statement 

(No -go  to  3.12) 

5.6  Staff  decides  if  plans  for 
branches/sequels 

6.4  Staff  decides  if  necessary  to 
reinitiate  OPP 

1.7.1  Comd  Issues  Initial  Guidance 

2.10  Comd's  Planning  Guidance 

3.11  Develop  New  COAs  Directed 
by  Comd  (as  required) 

(Yes  -  go  to  5.7) 

(Yes  -goto  1.0) 

1.7.2  Staff  Receives  Initial 

Guidance 

2.10.1  Comd  Provides  Planning 
Guidance 

3.12  Staff  Compare  Own  COAs 

(No  -  go  to  5.8) 

(No  -  go  to  6.5) 

1.8  Crisis  Action  Planning  or 

Supporting  Formations 
Known? 

2.10.2  Staff  Receive  Planning 
Guidance 

3.13  Staff  Wargame 

5.7  Develop  plans  for  branches 
and  sequels 

Reinitiate  OPP  as  required 

(Yes -go  to  1.9) 

3.0  COA  Development 

3.14  Staff  Identify  Branches  and 
Sequels 

5.8  Plan  wargame 

6.5  Update  and  issue  amendments 
as  required 

(No  -  Go  to  2.0) 

3.1  Staff  Review  Comd's  Planning 
Guide 

4.0  Decision 

5.9  Comd  decides  if  Plan  must  be 
submitted  to  higher 
authority  for  approval 

6.6  Prepare  and  issue  plans  as 
required 

1.9  Issue  Warning  Orders  to 

Subordinate/Supporting 

Formations 

3.2  Staff  Analysis 

4.1  Staff  Review 

Validation/Comparison 

Information 

(Yes  -  go  to  5. 10) 

Figure  13:  Overview  of  OPP  steps  performed 
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1.0  Situational  Awareness 

2.7.1  Staff  provides  Mission  Analysis  Briefing  to 
Comd 

3.4  Staff  Refine/Develop  COA 

4.9  Staff  develop  CONOP 

1.1  Awareness  of  Enemy  Situation 

2.7.2  Comd  Attends  Mission  Analysis  Briefing 

3.5  Skeleton  COA  given  to  LF 

4.10  Comd  and  Staff  present  CONOP  to 
Higher  Cmd 

1 .2  Awareness  of  Own  Situation 

2.8  Comd  Approves  Draft  Mission  Statement? 

4.0  COA  Validation 

4.11  Does  Higher  Cmd  approve  of 

CONOP? 

1.3  Awareness  of  Battlespace  Environment 

(No  -  return  to  2.6) 

4.1  COA  refined  enough  to  do  Staff 
Estimates? 

(No  -  return  to  4. 7) 

1 .4  Awareness  of  Other  Military  and  Civilian 
Resources 

(Yes  -  proceed  to  2.9) 

(No  -  return  to  3.0) 

(Yes  -  proceed  to  5.0) 

1 .5  Conduct  a  Meta-Cognitive  Check 

2.9  Initial  Intent 

(Yes  -  proceed  to  4.2) 

5.0  Plan  Development 

2.0  Mission  Analysis 

2.9.1  Comd  Provides  Initial  Intent  (CPG) 

4.2  Staff  Estimates 

5.1  Develop  Plan 

2.1  Mission  Receipt 

2.9.2  Staff  Receive  Initial  Intent  (CPG) 

4.3  Is  COA  viable? 

5.2  Comd  and  Staff:  Synchronization 
Wargame 

2.1.1  Mission  Receipt  from  Higher  Cmd 

3.0  COA  Development 

(No  -  inform  Comd  and  loop  back  to  2.0) 

5.3  Comd  decides  if  Plan  is  viable 

2.1.2  Comd  Initiates  Mission  Receipt 

3.1  Comd  has  Sufficient  Information  to  initiate 
COA  Development? 

(Yes  -  proceed  to  4.4) 

(No  -  return  to  5.1) 

2.1.3  LF  Informed  of  Mission  Receipt 

(No  -  return  to  2.0) 

4.4  Mental  Wargaming 

(Yes  -  proceed  to  5.4) 

2.2  Commander  and  Staff  Review  Own  and 

Higher  Factors,  Deductions, 
Assumptions,  Assigned  Tasks, 
Objectives 

(Yes  -  proceed  to  3.2) 

4.5  Staff  and  Comd  discuss  COA  viability 
(informal  IB) 

5.4  Develop  Contingency  Plans 

(Branches/Sequences) 

2.3  Planning  Guidance 

3.2  Skeleton  COA/Criteria  for  Success 

4.6  Comd  approves  validated  COA? 

5.5  Comd  and  Staff  Issue  Orders 

2.3.1  Commander  Provides  Initial  Planning 
Guidance 

Time  Constrained? 

(No  -  proceed  to  4. 7) 

5.5.1  LF  receive  Orders 

2.3.2  Staff  Receive  Initial  Planning  Guidance 

3.2.1  Comd  creates  Skeleton  COA/Criteria  for 
Success 

(Yes  -  proceed  to  4.8) 

5.6  Is  there  time  available? 

2.4  Staff  Develop  Information  Outlined  in  IPG 
and  Implied  Tasks 

3.2.2  Comd  asks  Staff  to  create  Skeleton 
COA/Criteria  for  Success 

4.7  Could  COA  be  tweaked  to  be 
validated? 

(No  -  End) 

2.5  Staff  Define  Most  Likely  ECOA 

3.2.2. 1  Staff  creates  Skeleton  COA 

(No  -  assume  new  COA  required,  return 
to  2.0) 

(Yes  -  proceed  to  5.7) 

2.6  Staff  Write  Draft  Mission  Statement 

3.3  Comd  Approves  of  COA/Criteria  for 

Success? 

(Yes  -  return  to  4.2) 

5.7  Comd,  Staff  and  LF  perform 

rehearsal  wargame  with  Sub 
Units 

2.7  Mission  Analysis  Briefing 

(No  -  return  to  3.1) 

4.8  Send  validated  COA  to  LF 

(Yes  -  proceed  to  3.4) 

Figure  14:  Overview  of  IOPP  steps  performed 

For  both  teams,  the  highest  level  steps  of  the  OPP  (i.e.  1.0,  2.0,  3.0)  were  performed  in  a  fairly 
linear  fashion  with  some  overlap  between  steps  1  and  2,  which  is  consistent  with  previous 
observations  (Bruyn,  Rehak,  Vokac  &  Lamoureux,  2005).  The  highest  level  IOPP  steps  (i.e. 

1.0,  2.0,  3.0)  were  not  performed  in  such  a  linear  fashion.  There  was  frequent  revisiting  of  step 
1.0  (Situational  Awareness)  throughout  planning  and  a  fair  amount  of  iteration  between  steps  2 
(Mission  Analysis)  and  3  (COA  development).  Specifically,  the  Yellow  and  Blue  teams  revisited 
step  1.0  nineteen  and  ten  times,  respectively.  Both  teams  switched  back  and  forth  between  steps 
2  and  3  three  times  throughout  the  planning  cycle.  This  reflected  the  way  in  which  the  IOPP  was 
designed  and  was  anticipated. 

Both  teams  identified  the  Criteria  For  Success  (CFS)  at  different  points  than  defined  in  the  IOPP. 
The  Blue  team  identified  the  CFS  earlier  in  the  process  than  outlined  (i.e.  during  presentation  of 
the  Commander’s  Planning  Guidance  or  CPG)  rather  than  at  the  time  when  the  skeleton  COA  is 
developed.  The  Yellow  team,  on  the  other  hand,  identified  CFS  later  in  the  process  than 
outlined.  Future  investigation  may  be  needed  with  respect  to  the  most  appropriate  time  to 
develop  the  CFS.  Developing  CFS  is  not  an  explicit  step  in  the  OPP  so  a  similar  observation 
cannot  be  made  for  the  OPP. 

A  significant  amount  of  repetition  between  mental  wargaming  and  develop/refine  COA  in  the 
IOPP.  Further,  it  was  observed  that  the  Blue  team  started  mental  wargaming  earlier  in  the 
process  than  defined  (i.e.  during  mission  analysis  rather  than  COA  Validation).  The  OPP  does 
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not  include  a  step  for  mental  wargaming,  however,  we  observed  that  both  teams  revisited  the 
refine  COA  step  following  the  staff  wargame  (traditional  rather  than  mental  wargaming). 

The  sequence  of  OPP  and  IOPP  steps  performed  during  each  experimental  condition  is  shown  in 
Annex  G.  The  number  of  steps  repeated  or  performed  out  of  order  was  calculated  (for  the 
highest  (i.e.  1.0,  2.0)  and  second  highest  level  (i.e.  1.1,  1.2,  1.3)  steps  only)  for  each  condition. 
A  step  was  considered  repeated  if  it  was  performed  at  least  twice  during  the  planning  session. 
Table  9  shows  that,  for  the  Blue  team,  the  number  of  repeated  steps  was  slightly  higher  for  the 
IOPP  than  the  OPP  at  the  highest  level  of  decomposition.  At  the  second  highest  level  of 
decomposition,  we  observed  that  the  Blue  team  repeated  steps  within  the  OPP  slightly  more  than 
the  IOPP.  The  Yellow  team,  on  the  other  hand,  repeated  steps  within  the  OPP  slightly  more  than 
when  using  the  IOPP  at  the  highest  level  of  decomposition.  At  the  second  highest  level  of 
decomposition  the  Yellow  team  repeated  steps  slightly  more  often  when  using  the  IOPP  than  the 
OPP.  It  was  expected  that  there  would  be  more  repetition  when  teams  used  the  IOPP  given  that 
it  was  designed  to  be  more  iterative,  especially  with  respect  to  revisiting  SA  continuously 
throughout  the  process,  and  the  fact  that  Mission  Analysis  and  COA  Development  were  intended 
to  be  performed  concurrently.  However,  our  results  did  not  support  this. 


Table  9:  Number  of  OPP  and  IOPP  steps  repeated 


IOPP  Yellow  Team 

IOPP  Blue  Team 

OPP  Blue  Team 

OPP  Yellow  Team 

Highest  Decomposition  (e.g. 
1.0, 2.0) 

2 

4 

2 

3 

Second  Highest 
Decomposition  (e.g.  1.1, 1.2) 

10 

6 

7 

9 

The  amount  of  looping  that  was  performed  for  both  the  OPP  and  IOPP  was  also  calculated.  A 
loop  was  defined  as  an  instance  in  which  the  planning  cycle,  having  been  moving  forward 
through  the  sequence  of  functions,  went  back  to  an  earlier  function  in  the  process  sequence, 
whether  that  step  had  previously  been  performed  or  not.  Table  10  and  Table  11  below  show  the 
amount  of  looping  for  the  OPP  and  IOPP  at  both  the  highest  and  second  highest  level  of 
decomposition.  For  the  OPP,  looping  occurred  most  frequently  on  the  first  day  between  steps  1 
and  2  (i.e.  Initiation  and  Orientation)  at  the  highest  level  of  decomposition  and  repeated  loops 
back  to  1.4  (gather  planning  tools)  and  1.6  (Commander  makes  initial  assessment)  at  the  second 
highest  level.  This  is  not  surprising  given  that  the  team  was  required  to  create  their  own  maps 
and  other  planning  tools,  which  required  a  significant  amount  of  time.  However,  they  could  not 
afford  to  delay  the  progress  of  the  planning  so  continued  through  the  OPP  while  regularly 
revisiting  step  1.4.  Likewise,  the  Commander  took  a  long  time  to  perform  his  initial  assessment 
but,  based  on  a  comment  by  the  Commander,  did  not  want  to  keep  the  team  from  progressing 
through  the  planning  process.  Therefore,  step  1.6  was  frequently  revisited. 

Looping  by  teams  performing  the  IOPP  occurred  most  frequently  among  Steps  1  (situational 
awareness),  2  (mission  analysis)  and  3  (COA  development)  at  the  highest  level  of  decomposition 
and  repeated  loops  back  to  steps  1.3  (awareness  of  battlespace  environment)  and  1.4  (awareness 
of  other  military  and  civilian  resources)  at  the  second  highest  level.  This  frequent  looping  among 
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steps  1,  2  and  3  was  anticipated  because  the  IOPP  mandates  continuous  situational  awareness, 
and  second,  it  was  prescribed  that  mission  analysis  and  COA  development  be  performed 
concurrently.  It  does  not  appear  that  there  was  a  practice  effect  for  looping  during  the  planning 
processes.  If  the  teams  benefited  from  a  practice  effect,  then  it  would  be  expected  that  the  team 
would  perform  less  looping  on  day  2  than  day  1  as  they  become  more  familiar  with  the  planning 
exercise  itself.  Further,  there  would  be  a  similar  pattern  for  both  the  Blue  and  Yellow  teams. 
The  Yellow  team  were  observed  to  loop  more  using  the  IOPP  on  day  1  than  on  day  2  with  the 
OPP.  However,  the  Blue  team  looped  back  and  forth  between  functions  of  the  IOPP  (performed 
on  day  2)  compared  to  the  OPP  (performed  on  day  1).  If  there  was  a  practice  effect,  we  would 
not  expect  to  see  such  an  increase  in  the  amount  of  looping.  Hence,  these  results  do  not  suggest 
the  existence  of  a  practice  effect. 


Table  10:  Frequency  of  loops  for  OPP  and  IOPP 


IOPP  Yellow  Team 

IOPP  Blue  Team 

OPP  Blue  Team 

OPP  Yellow  Team 

Highest  Decomposition  (e.g. 
10,2.0) 

22 

19 

4 

3 

Second  Highest 
Decomposition  (e.g.  1.1, 1.2) 

26 

26 

13 

9 

Table  1 1 :  Incidence  of  loops  for  OPP  and  IOPP 


IOPP  Yellow  Team 

IOPP  Blue  Team 

OPP  Blue  Team 

OPP  Yellow  Team 

Highest  Decomposition  (e.g. 
10,2.0) 

26.8% 

31.1% 

9.1% 

6.5% 

Second  Highest 
Decomposition  (e.g.  1.1, 1.2) 

31.7% 

42.6% 

29.5% 

19.6% 

4.2.7  Number  of  non-mandated  steps  added  to  the  planning  process 

Both  teams  added  non-mandated  steps  to  the  IOPP.  Given  that  this  experiment  was  the  first  time 
an  experienced  team  had  applied  the  IOPP,  steps  that  were  skipped  or  added  are  of  particular 
interest  for  further  investigation.  First,  the  Commander  of  the  Yellow  team  added  a  progress 
check  step  to  the  IOPP,  likely  because  the  team  was  unfamiliar  with  the  IOPP  and  specific  time 
constraints  were  imposed  on  the  teams.  The  Blue  team  developed  a  back-up  or  alternative  COA, 
even  though  the  IOPP  prescribes  the  development  of  a  single  COA.  Nonetheless  the  Blue  team 
reported  that  they  felt  the  need  to  at  least  have  a  second  COA  on  hand  that  could  be  further 
developed.  Comments  by  the  participants  during  the  AARs  indicate  that  they  experienced  some 
discomfort  with  developing  a  single  COA.  Specifically,  participants  stated  that  they  would 
require  a  certain  level  of  certainty  before  committing  to  a  single  COA,  and  they  felt  that  they  did 
not  have  this  level  of  certainty. 
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4.2.8  Total  errors  made  in  planning  process 

It  was  intended  that  observers  identity  errors  that  teams  made  in  applying  the  planning  processes 
(as  opposed  to  errors  in  the  planning  products).  However,  it  became  apparent  that  observers 
classified  errors  as  steps  in  the  planning  process  that  were  skipped  or  abbreviated  or  non- 
mandated  steps  that  were  added  to  the  process.  This  data  was  collected  and  analyzed  separately 
and  is  reported  in  earlier  sections  of  the  report  (see  sections  4.2.6  and  4.2.7).  As  such,  data 
collected  from  observers  did  not  facilitate  an  analysis  of  the  total  errors  made  in  the  planning 
process. 

4.2.9  Requests  for  additional  information  or  clarification 

The  SME  observers  created  the  scenarios  as  well  as  the  Order  passed  down  from  Higher 
Command  and  therefore  had  a  lot  of  background  knowledge.  They  also  occasionally  took  on  the 
role  of  higher  commander  if  the  teams  felt  that  they  required  clarification  or  further  information 
relating  to  the  orders  that  were  passed  along  to  them.  The  content  of  the  requests  for  additional 
information  or  clarification  were  noted  by  observers  during  the  planning  exercises.  Teams  using 
the  OPP  generally  asked  questions  relating  to  assumptions,  relevant  factors,  timings,  composition 
of  a  battle  group,  details  of  the  ORB  AT,  OPP  terminology  and  the  process  of  staff  analysis.  This 
may  be  due  to  the  limited  experience  of  the  participants  in  applying  the  OPP.  Questions  relating 
to  assumptions  and  details  of  the  battlespace  environment  suggest  that  either  more  detail  should 
be  provided  in  the  scenario  materials  and  the  orders  from  higher  command,  or  that  an  SME 
playing  the  role  of  higher  command  should  be  present  in  this  type  of  experiment. 

Teams  using  the  IOPP  asked  questions  relating  to  the  process  itself  (e.g.  how  to  outlined  a  COA 
and  details  of  the  enemy  COA),  expectations  for  the  final  products,  IOPP  terminology, 
assumptions,  and  information  about  the  battlespace  environment.  Given  that  this  is  the  first 
exposure  that  both  teams  had  to  the  IOPP,  questions  relating  to  the  IOPP  itself  were  expected.  It 
may  also  be  that  the  online  training  did  not  provide  sufficient  training  in  how  to  apply  the  IOPP. 
Similar  to  the  OPP,  questions  relating  to  assumptions  and  details  of  the  battlespace  environment 
suggest  that  either  more  detail  should  be  provided  in  the  scenario  materials  and  the  orders  from 
higher  command,  or  that  an  SME  playing  the  role  of  higher  command  should  be  present  in  this 
type  of  experiment. 

4.2.10  Number  of  team  briefings  observed  -  both  formal  and  informal 

Observers  were  to  collect  data  on  the  number  of  team  briefings,  both  formal  and  informal,  as 
well  as  the  number  of  participants  attending  the  briefings.  It  was  observed,  however,  that  teams 
worked  very  collaboratively,  especially  when  using  the  IOPP,  making  it  impossible  to  distinguish 
and  enumerate  separate  briefings  in  a  meaningful  way. 

4.2.1 1  Number  of  options  presented  to  commander 

The  Yellow  team  developed  and  presented  three  friendly  CO  As  to  the  Commander  when  using 
the  OPP  and  only  one  friendly  COA  when  using  the  IOPP.  Hence,  in  both  cases,  the  Yellow 
team  developed  the  number  of  COAs  prescribed  by  the  respective  planning  process.  In  contrast, 
the  Blue  team  developed  and  presented  two  COAs  to  the  Commander  when  following  both  the 
OPP  and  IOPP.  The  blue  team  indicated  that  they  were  only  able  to  develop  two  COAs  when 
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using  the  OPP  due  to  time  constraints.  When  the  team  followed  the  IOPP,  they  developed  one 
main  COA  as  well  as  one  back-up  COA.  As  previously  mentioned,  this  was  likely  an  indication 
of  the  team’s,  or  more  likely  the  Commander’s,  lack  of  comfort  in  developing  and  committing  to 
a  single  COA. 

4.2.12  Quality  of  planning  process  and  products 

The  SME  observers  rated  the  quality  of  the  planning  process  and  products.  Each  SME  observer 
was  assigned  to  a  single  team  and  was  therefore  able  to  observe  the  application  of  both  the  OPP 
and  the  IOPP  by  the  same  team.  SME  observers  rated  the  quality  of  the  following  steps  in  the 
planning  process: 

•  Mission  analysis; 

•  Commander’s  guidance; 

•  Staff  analysis;  and 

•  COA  Evaluation/ Validation. 

They  also  rated  the  overall  efficiency  of  team  in  applying  the  planning  process. 

SME  observers  also  rated  the  quality  of  the  following  planning  products: 

•  General  working  documents; 

•  Selected  COA; 

•  CONOP; 

•  Plan;  and 

•  Order. 

Figure  15  to  Figure  17  show  the  SME  ratings  of  the  planning  process  and  products  for  both  the 
OPP  and  IOPP.  Figure  15  and  Figure  16  show  the  results  for  individual  SME  observers,  whereas 
Figure  17  shows  the  overall  ratings  averaged  across  observers.  A  five-point  rating  scale  was 
used  where  0  represented  ‘poor’  and  5  represented  ‘exceptional’.  It  is  important  to  note  that,  for 
the  Yellow  team,  the  SME  observer  considered  the  CONOP  equivalent  to  the  Plan  and  therefore 
did  not  rate  the  quality  of  the  CONOP  separately  from  that  of  the  plan. 


Human systems 


Maple  Skies  Exercise  Management  Process 


Page  41 


HUMANST5r£  MS 

IMCWPO rttd/d 


Figure  15  :  Yellow  Team  SME  ratings  of  planning  process  and  products 
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Figure  16  :  Blue  Team  SME  ratings  of  planning  process  and  products 
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Figure  17  :  Combined  SME  ratings  of  planning  process  and  products 

The  SME  observer  of  the  yellow  team  generally  rated  the  quality  of  IOPP  planning  process  and 
products  equivalent  to  those  of  the  OPP,  with  the  exception  of  the  Staff  Analysis  process,  which 
was  reportedly  of  higher  quality  when  the  IOPP  was  used,  the  overall  efficiency  of  the  team, 
which  was  reportedly  higher  when  the  OPP  was  used,  and  the  quality  of  the  CONOP.  The 
CONOP  produced  using  the  IOPP  was  considered  equivalent  to  the  Plan  and  was  therefore  not 
rated  separately.  The  SME  observer  of  the  blue  team,  in  contrast,  consistently  rated  the  quality 
of  the  planning  process  and  all  planning  products  higher  for  the  IOPP  than  the  OPP. 


4.2.13  Usability  of  planning  processes 

Annex  H  includes  the  individual  participant  ratings  of  the  ease  of  use,  learnability,  efficiency, 
effectiveness,  complexity,  suitability  and  overall  user  satisfaction  for  both  the  OPP  and  IOPP. 
Ratings  for  both  teams  were  combined  to  calculate  mean,  minimum,  maximum  and  standard 
deviation  scores  for  all  usability  measures  (Table  12). 


Table  12:  Participant  usability  ratings  of  OPP  and  IOPP 


Usability  of  Planning 
Process 

Max 

Min 

Mean 

Std.  Dev 

Mean  + 
1SD 

Mean  - 
1SD 

IOPP 

4.50 

2.50 

3.73 

0.83 

4.56 

2.91 

Ease  of  use 

OPP 

3.00 

2.50 

2.70 

0.27 

2.97 

2.43 

IOPP 

4.00 

2.50 

3.65 

0.68 

4.33 

2.97 

Learnability 

OPP 

3.50 

2.00 

2.80 

0.64 

3.44 

2.16  | 

IOPP 

4.50 

4.00 

4.10 

0.20 

4.30 

3.90 

Efficiency 

OPP 

4.00 

2.00 

3.30 

0.87 

4.17 

2.43 

IOPP 

4.50 

3.00 

3.72 

0.63 

4.35 

3.08 

Effectiveness 

OPP 

4.00 

3.00 

3.50 

0.55 

4.05 

2.95 
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IOPP 

4.50 

2.00 

3.55 

0.93 

4.48 

2.62 

Complexity 

OPP 

3.00 

2.00 

2.20 

0.45 

2.65 

1.75 

IOPP 

4.50 

2.00 

3.77 

0.98 

4.75 

2.78 

Suitability 

OPP 

4.50 

2.00 

3.40 

1.03 

4.43 

2.37 

Overall  user 

IOPP 

5.00 

4.00 

4.18 

0.41 

4.59 

3.78 

satisfaction 

OPP 

5.00 

2.00 

3.50 

1.24 

4.74 

2.26  | 

On  average,  the  IOPP  scored  higher  than  the  OPP  with  respect  to  ease  of  use,  learnability, 
efficiency,  effectiveness,  complexity  (i.e.  the  IOPP  was  less  complex),  suitability  and  overall 
user  satisfaction. 


4.2.14  User  trust  and  reliability  of  planning  processes 

Participants  rated  their  level  of  trust  in  the  OPP  and  IOPP  to  create  a  quality  Plan.  Given  that 
user  trust  is  often  related  to  reliability,  participants  were  also  asked  to  rate  the  reliability  of  Plans 
created  by  the  OPP  and  IOPP  to  accomplish  the  mission.  In  other  words,  we  asked  participants 
to  rate  how  reliable  they  think  the  OPP  and  IOPP  are  in  generating  successful  Plans  that  will 
accomplish  the  mission.  Research  suggests  that  reliability  is  only  one  factor  affecting  one’s  trust 
in  a  person  or  system  (Adams,  Bruyn,  Houde  &  Angelopolous,  2003)  and  therefore  separate 
measures  of  trust  and  reliability  were  included. 

Individual  participant  ratings  of  trust  and  reliability  of  the  OPP  and  IOPP  to  create  a  quality  plan 
are  shown  in  Annex  H.  Table  13  shows  the  mean  user  trust  and  reliability  rating  across 
participants,  as  well  as  the  minimum  and  maximum  ratings  and  standard  deviations.  Results 
suggest  that  the  participants  placed  a  slightly  higher  level  of  trust  in  the  OPP  than  the  IOPP  to 
create  a  good  plan.  Likewise,  participants  rated  the  OPP  as  more  reliable  than  the  IOPP  in  terms 
of  creating  a  good  plan.  This  is  not  surprising  given  that,  prior  to  this  experiment,  none  of  the 
participants  had  any  experience,  either  in  training  or  in  operations,  with  the  IOPP  and  therefore 
had  no  perception  of  its  reliability. 


Table  13:  User  trust  and  reliability  of  OPP  and  IOPP 


Trust  and  Reliability  of  Planning 
Process 

Max 

Min 

Mean 

Std.  Dev 

Mean  + 
1SD 

Mean  - 
1SD 

IOPP 

5.00 

2.50 

3.70 

0.84 

4.54 

2.86 

User  trust/confidence  OPP 

4.50 

3.00 

3.90 

0.72 

4.62 

3.18 

IOPP 

4.50 

2.75 

3.68 

0.70 

4.37 

2.98 

Reliability  OPP 

4.50 

3.50 

4.10 

0.42 

4.52 

3.68 

4.2.15  User  trust  and  reliability  in  planning  products 

In  each  session,  participants  rated  their  level  of  trust  in  the  overall  plan,  to  accomplish  the 
mission.  As  noted  above,  reliability  is  often  related  to  user  trust  so  participants  also  rated  the 
reliability  of  the  plans  (created  by  both  the  OPP  and  IOPP)  to  accomplish  the  mission. 

Individual  participant  ratings  of  trust  and  reliability  of  plans  created  by  both  the  OPP  and  IOPP 
to  successfully  complete  the  mission  are  shown  in  Annex  H.  Table  14  shows  the  mean  rating 
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across  participants,  as  well  as  the  minimum  and  maximum  ratings  and  standard  deviations. 
Results  suggest  that  the  participants  placed  a  slightly  higher  level  of  trust  in  plans  created  using 
the  OPP  rather  than  the  IOPP  in  terms  of  accomplishing  the  mission.  Likewise,  participants 
rated  plans  created  using  the  OPP  more  reliable  in  terms  of  accomplishing  the  mission,  compared 
to  plans  created  using  the  IOPP.  This  is  not  surprising  given  that,  prior  to  this  experiment,  none 
of  the  participants  had  any  experience,  either  in  training  or  in  operations,  with  the  IOPP  and 
therefore  had  no  perception  of  its  reliability  in  terms  of  creating  plans  that  accomplish  the 
mission. 


Table  14:  User  trust  and  reliability  of  planning  products 


Trust  and  Reliability  in 
Planning  Products 

Max 

Min 

Mean 

Std. 

Dev 

Mean  + 
1SD 

Mean  - 
1SD 

User  trust/confidence  in 

IOPP 

5.00 

3.50 

4.07 

0.63 

4.70 

3.43 

Plan 

OPP 

4.50 

3.00 

3.80 

0.63 

4.43 

3.17 

IOPP 

5.00 

3.50 

4.08 

0.52 

4.60 

3.56 

OPP 

4.50 

3.00 

4.00 

0.64 

4.64 

3.36 

Reliability  of  Plan 

OPP 

4.50 

3.00 

3.90 

0.58 

4.48 

3.32 

4.2.16  Accuracy  and  overall  quality  of  Plan  created  by  OPP  and  IOPP 

For  both  OPP  and  IOPP  conditions,  participants  rated  the  accuracy  of  the  overall  plan  in 
completing  the  mission  as  well  as  the  overall  quality  of  the  plan.  The  accuracy  of  the  plan  in 
completing  the  mission  can  be  defined  as  how  precise  the  plan  is  in  addressing  the  mission.  On 
the  other  hand,  the  overall  quality  of  the  plan  may  include  additional  considerations  such  as  how 
easily  it  can  be  understood  by  subordinate  commanders.  Participants’  ratings  of  accuracy  and 
overall  quality  of  the  plan  are  shown  in  Annex  H.  Table  15  shows  the  mean  rating  across 
participants,  as  well  as  the  minimum  and  maximum  ratings  and  standard  deviations.  The 
participants  generally  rated  the  accuracy  of  the  overall  plans  created  with  the  OPP  and  IOPP  as 
equivalent.  The  average  rating  of  the  overall  quality  of  the  plan  was  slightly  higher  when  the 
plan  was  created  using  the  IOPP  compared  to  the  OPP. 


Table  15:  Accuracy  and  overall  quality  of  Plan 


Accuracy  and  Overall 
Quality  of  Plan 

Max 

Min 

Mean 

Std. 

Dev 

Mean  + 
1SD 

Mean  - 
1SD 

IOPP 

4.50 

3.00 

3.82 

0.58 

4.40 

3.24 

Accuracy  of  Plan 

OPP 

4.50 

3.00 

3.90 

0.58 

i  4.48 

3.32 

IOPP 

4.00 

3.00 

3.63 

0.53 

4.17 

3.10 

Overall  quality  of  Plan 

OPP 

4.50 

2.50 

3.60 

0.87 

4.47 

2.73 

4.2.17  Accuracy  of  planning  process 

Participants  rated  the  OPP  and  IOPP  in  terms  of  their  potential  to  lead  to  errors  and  the  potential 
criticality  of  errors  that  might  result.  Again,  individual  participant  ratings  are  provided  in  Annex 
H.  Table  16  shows  the  mean  rating  across  participants,  as  well  as  the  minimum  and  maximum 
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ratings  and  standard  deviations.  In  this  case,  a  rating  of  1  represented  a  very  low  potential  to 
lead  to  errors,  whereas  a  rating  of  5  represented  a  very  high  potential  to  lead  to  errors. 
Participants  rated  the  IOPP  as  having  slightly  more  potential  to  lead  to  errors  in  the  overall  plan 
compared  to  the  OPP.  Likewise,  they  rated  the  IOPP  as  being  slightly  more  likely  to  lead  to  a 
plan  that  contains  critical  errors  than  the  OPP. 


Table  16:  Ratings  of  potential  of  planning  processes  to  lead  to  errors  in  Plan  and 

criticality  of  errors 


Accuracy  of  Planning 
Process 

Max 

Min 

Mean 

Std. 

Dev 

Mean  + 
1SD 

Mean 

1SD 

IOPP 

4.00 

2.00 

3.35 

0.87 

4.22 

2.48 

Potential  to  lead  to  errors 

OPP 

4.00 

2.00 

3.40 

0.89 

4.29 

2.51 

IOPP 

4.00 

2.00 

3.15 

0.78 

3.93 

2.37 

Potential  criticality  of  errors 

OPP 

4.00 

2.50 

3.40 

0.72 

|  4.12 

2.68 

4.2.18  Operational  effectiveness  of  planning  process 

Participants  were  asked  to  rate  the  operational  effectiveness  (i.e.  the  suitability  of  the  process  in  a 
variety  of  operational  contexts)  of  both  the  OPP  and  IOPP.  Participants  also  rated  how  well  their 
team  followed  each  process  during  the  experiment  as  a  secondary  measure  of  operational 
effectiveness.  Individual  participant  ratings  are  provided  in  Annex  H.  Table  17  shows  the  mean 
rating  across  participants,  as  well  as  the  minimum  and  maximum  ratings  and  standard  deviations. 
Results  suggest  that  they  found  the  IOPP  slightly  more  suitable  in  a  variety  of  operational 
contexts  compared  to  the  OPP.  However,  the  participants  reported  that  the  team  was  better  at 
following  the  OPP  compared  to  the  IOPP  during  the  experiment.  Given  that  the  teams  are  more 
familiar  with  the  OPP,  it  is  not  surprising  that  they  felt  that  they  were  more  able  to  follow  the 
process  compared  to  the  IOPP. 


Table  17:  Operational  effectiveness  of  OPP  and  IOPP 


Operational  Effectiveness 
of  Planning  Process 

Max 

Min 

Mean 

Std.  Dev 

Mean  + 
1SD 

Mean  - 
1SD 

Suitability  in  a  variety  of 

IOPP 

5.00 

2.50 

3.72 

0.89 

4.61 

2.82 

operational  contexts 

OPP 

5.00 

2.50 

3.60 

0.99 

|  4.59 

2.61 

IOPP 

4.00 

3.50 

3.75 

0.27 

4.02 

3.48 

Team  followed  process 

OPP 

4.50 

3.00 

3.90 

0.58 

|  4.48 

3.32 

4.2.19  Workload  ratings 

Participants  completed  the  NASA  TLX  workload  measure  twice  for  each  experimental 
condition;  halfway  through  the  day  and  at  the  end  of  the  day.  Individual  participant  workload 
ratings  for  both  planning  processes  are  provided  in  Annex  I.  Figure  18  shows  the  average 
workload  rating  for  each  team  using  the  OPP  and  IOPP.  Both  teams  reported  higher  workload 
levels  on  the  first  day  compared  to  the  second  day  of  the  exercise.  For  the  Blue  team,  this 
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meant  that  lower  workload  was  associated  with  the  IOPP  than  OPP,  whereas  for  the  Yellow 
team  lower  workload  levels  were  associated  with  the  OPP  than  IOPP.  This  trend  is  most  likely 
the  result  of  a  learning  effect  as  both  teams  became  more  accustomed  to  the  planning  exercise 
over  the  course  of  the  experimental  exercise.  Comments  of  the  Yellow  team  during  the  AAR  at 
the  end  of  the  first  day  support  this  interpretation.  The  team  expressed  the  opinion  that  they 
spent  too  much  time  familiarizing  themselves  with  the  scenario  before  they  started  planning 
because  they  were  not  familiar  with  things  such  as  the  battlespace.  Therefore  they  spent  a  lot  of 
time  studying  the  map  and  orienting  themselves.  In  addition,  the  Commander  of  the  Yellow 
team  commented  that  he  thought  he  “only  hit  1/3  of  the  [IOPP}  steps”  and  felt  the  team  spent 
too  much  time  in  mission  analysis. 


Average  workload  for  participants 


IOPP,  Day  1,  IOPP,  Day  2,  OPP,  Day  1,  OPP,  Day  2, 
Yellow  Blue  Blue  Yellow 


Figure  18  :  Overall  workload  levels  for  both  teams  using  the  OPP  and  IOPP 

Figure  19  shows  the  overall  workload  ratings,  collapsed  across  teams,  for  the  OPP  and  IOPP. 
The  graph  shows  that  the  average  workload  rating  was  only  slightly  lower  when  teams  used  the 
IOPP  compared  to  the  OPP. 
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OPP  workload  vs.  IOPP  workload 
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Figure  19  :  Overall  workload  rating  for  OPP  and  IOPP 

Figure  20  shows  that  the  average  workload  was  higher  at  the  end  of  the  day  than  the  middle  of 
the  day.  This  is  not  surprising  given  that  teams  were  able  to  adjust  their  schedule,  or  “battle 
rhythm,”  earlier  in  the  day  but  faced  a  “hard”  deadline  at  the  end  of  the  day  for  the  presentation 
of  orders.  Their  perceived  workload  likely  increased  toward  as  teams  worked  to  meet  this 
deadline. 


Workload  after  lunch  and  end  of  day 
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Figure  20  :  Overall  workload  measure  after  lunch  and  at  the  end  of  the  day 
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4.2.20  Other  relevant  observations 

Many  of  the  observations  below  came  from  SME  or  observer  notes  during  the  planning  exercise 
as  well  as  the  outcome  of  the  daily  AARs. 

4. 2. 20. 1  Collaboration 

In  general,  significantly  more  collaboration  between  the  Commander  and  his  staff  occured  when 
the  team  used  the  IOPP  than  the  OPP.  This  was  especially  evident  for  the  blue  team.  They 
organized  their  planning  room  in  a  more  centralized  fashion  on  the  second  day  when  the  IOPP 
was  applied.  Specifically,  all  of  their  working  products  were  in  common  areas,  there  was  a 
common  area  in  which  the  team  collaborated  and  held  briefings,  and  they  engaged  in  more 
collaboration  when  setting  the  battle  rhythm. 

Both  teams  performed  several  of  the  IOPP  steps  together  as  a  group.  For  example,  the  skeleton 
COA,  while  conceived  by  the  Commander,  was  developed  collectively  by  Commander  and  Staff. 
In  contrast,  when  following  the  OPP,  the  teams  tended  to  perform  steps  more  independently.  In 
the  AARs,  participants  also  noted  that  the  IOPP  worked  well  because  errors  were  spotted 
immediately  because  of  more  collaboration  and  more  effective  communication. 

4.2.20.2  Shared  mental  models 

During  the  AARs  participants  noted  that  they  were  surprised  how  much  the  team  was  “on  the 
same  page”  when  using  the  IOPP.  In  fact,  the  blue  team  estimated  that  the  Commander  and  Staff 
were  on  the  same  page  for  95%  of  the  time.  Participants  also  liked  that  the  IOPP  encourages 
conversation  and  brainstorming  through  constant  wargaming. 


4.2.20.3  Commander  involvement 

Participants  suggested  that  the  personality  of  the  Commander  would  play  a  role  with  the  IOPP 
more  so  than  with  the  OPP.  Although  participants  did  not  provide  further  detail  on  why  the 
Commander’s  personality  would  play  more  of  a  role  with  the  IOPP,  it  is  suspected  that  it  is 
because  the  identification  of  a  single  COA  is  based  on  the  experience  of  the  Commander.  The 
Commander  on  the  Yellow  team  liked  that  he  had  the  ability  to  identify  a  COA  early  on  rather 
than  the  staff  developing  three  COA  from  which  he  would  select  one. 

4.2.20.4  Training 

During  in  the  AARs  participants  emphasized  that  people  should  be  familiar  with  OPP  before 
learning  the  IOPP.  The  Commander  of  the  Yellow  team  stated  that  if  adopted  by  CF,  he  “sees  a 
potential  problem  with  people  who  will  only  learn  the  IOPP”.  Although  he  does  think  the  IOPP 
would  work  well  for  those  that  are  fully  trained  in  the  OPP. 

4. 2. 20. 5  Situ  a  tional  A  wareness 

Both  teams  recommended  a  list  of  “bullet-proof  questions”  for  each  element  of  SA  (especially  the 
meta-cognitive  check).  They  felt  that  SA  and  its  application  were  too  loosely  defined  in  the 
current  IOPP.  Furthermore,  the  term  meta-cognitive  was  not  popular  with  the  participants.  Both 
teams  liked  the  idea  of  a  met-cognitive  check  but  indicated  that  they  would  prefer  different 
terminology. 
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Both  teams  noted  that  the  static  scenarios  used  for  this  experiment  did  not  rigorously  test  their 
capability  to  maintain  situational  awareness.  Given  that  things  were  not  changing  in  the 
operational  environment,  there  was  no  new  incoming  information  on  own  or  enemy  situation, 
which  dramatically  limited  the  requirements  for  maintaining  SA. 

Again,  both  teams  also  felt  that  someone  in  the  staff  should  be  in  charge  of  situational  awareness, 
ensuring  that  it  is  performed  regularly  throughout  the  planning  process.  One  team  also  suggested 
a  sixth  component  of  situational  awareness,  namely  a  check  of  whether  the  COA/Plan  has  been 
tweaked. 

4.2.20.6  Applicability  at  Battle  Group  level 

Both  teams  expressed  the  opinion  that  the  IOPP  may  be  more  appropriate  at  Brigade/Division 
level,  as  opposed  to  the  battle  group  level.  This  is  somewhat  surprising  in  that  time  scales  are 
longer  and  resources  are  more  abundant  at  higher  levels  and  that  a  planning  process  based  on 
intuitive  decision  making  is  likely  to  be  more  advantageous  at  a  level  where  time  and  resources 
are  scarce  (Bruyn  et  al.,  2005).  Furthermore,  one  of  the  SMEs  found  it  odd  that  the  participants 
would  suggest  that  applicability  of  the  IOPP  exists  at  the  Brigade  or  Division  level  when  the 
complexity  of  problems  typically  faced  at  the  Brigade  level  and  higher  tend  to  require  a  more 
formal  process. 

4.2.20.7  Application  of  the  OPP 

To  a  lesser  degree,  participants  had  some  comments  regarding  the  application  of  the  OPP  in  an 
operational  environment.  First,  it  was  noted  that  a  planning  staff  will  typically  skip  or  abbreviate 
steps  in  the  OPP  as  time  pressure  increases.  For  example,  the  staff  may  develop  two  rather  than 
three  own  COAs  or  they  may  only  consider  the  most  likely  ECOA  as  opposed  to  the  most  likely 
and  most  dangerous.  This  is  consistent  with  observations  made  in  previous  studies  (Bruyn  et  al., 
2005). 

4.2.20.8  Benefits  and  limitations  of  the  IOPP 

During  the  AAR  at  the  end  of  the  second  day,  both  teams  were  asked  to  discuss  both  the  benefits 
and  limitations  of  the  IOPP.  The  teams  noted  the  following  benefits  of  the  IOPP  relative  to  the 
OPP: 

•  Lower  formations  play  a  part  earlier  in  the  process; 

•  There  is  more  collaboration  between  the  Commander  and  Staff; 

•  The  process  is  more  efficient,  which  is  especially  good  for  crisis  planning; 

•  There  is  more  time  to  devote  to  detail  of  COA; 

•  Take  advantage  of  the  experience  of  the  Commander; 

•  A  COA  is  identified  earlier; 

•  It  is  not  too  process-  or  product-oriented;  and 

•  There  is  more  frequent  formal  communication  up  and  down  the  chain  of  command. 

In  terms  of  limitations  of  the  IOPP,  the  teams  made  the  following  points: 

•  There  is  a  risk  of  committing  to  a  bad  COA  too  soon; 

•  It  is  not  a  “good  process  for  the  socially  inept  or  intellectually  weak  because  it  uses 
instinct”; 

•  It  requires  a  strong  leader  to  control  discussions; 
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•  Personality  issues  with  the  Commander  may  become  an  issue; 

•  A  process  involving  a  lot  of  brainstorming  isn’t  typical  in  the  military  and  may  be 
difficult  to  adopt; 

•  There  are  fewer  “benchmarks”  on  the  way  to  the  Plan; 

•  It  is  unclear  how  well  the  IOPP  will  work  in  asymmetric  environment; 

•  The  process  relies  on  intellect  and  experience  of  people  involved; 

•  There  is  too  much  pressure  to  focus  on  situational  awareness,  which  makes  it  easier  to 
get  off  track;  and 

•  The  process  requires  discipline  to  focus  on  situational  awareness. 

The  number  of  benefits  identified  by  participants  is  encouraging  and  certainly  supports  future 
iterations  and  evaluation  of  the  current  IOPP.  Many  of  the  limitations  could  be  accounted  for  by 
the  participants’  lack  of  familiarity  and  therefore  level  of  comfort  with  the  IOPP.  However, 
other  limitations,  such  as  the  existence  of  fewer  benchmarks  or  the  applicability  of  the  IOPP  in 
an  asymmetric  environment,  represent  opportunities  for  improvements  in  the  IOPP  and  should  be 
considered  in  future  work. 
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5.  Conclusions  and  recommendations 


5.1  IOPP  Model 

The  first  iteration  of  the  IOPP  model  exhibits  the  ‘best’  characteristics  of  other  intuitive  planning 
models  and  also  incorporates  findings  from  previous  work  investigating  application  of  the  OPP  in 
the  CF  (Bruyn  et  al.,  2005).  The  key  characteristics  of  the  IOPP  are: 

•  The  development  of  only  1  CO  A  at  a  time; 

•  Mental  wargaming  as  opposed  to  traditional  wargaming; 

•  Continuous  situational  awareness  throughout  the  process; 

•  Feedback  loops  from  COA  development,  COA  validation  and  Plan  Development  to  the 
beginning  of  Mission  Analysis; 

•  The  concurrent  performance  of  Mission  Analysis  and  COA  Development; 

•  The  continuous  reciprocal  input  and  feedback  from  SA  to  every  other  high  level  function 
in  the  IOPP; 

•  Early  and  frequent  communication  with  lower  formations;  and 

•  A  time  scale  across  the  bottom  to  show  functions  against  the  progression  of  time,  with 
the  proportions  of  each  step  representing  the  approximate  amount  of  time  that  it  is 
anticipated  that  a  planning  team  will  spend  on  each  step. 

The  IOPP  improves  on  existing  intuitive  models  of  planning  by  including  SA  checks  and  iterative 
cycles.  The  recurring  SA  checks  and  iteration  between  mission  analysis  and  COA  development 
appears  to  represent  a  closer  match  to  the  way  in  which  planning  that  is  performed  by  CF 
planning  staffs  at  the  Bde  level  (Bruyn  et  al.,  2005). 

Although  the  IOPP  has  many  characteristics  that  differ  from  the  OPP,  a  conscious  effort  was 
made  to  maintain  much  of  the  same  terminology  (e.g.  Comd’s  intent,  wargaming,  mission 
analysis,  COA  development)  as  well  as  some  of  the  outputs  generated  (e.g.  mission  statement), 
planning  tools  (e.g.  tasking  matrix),  and  formal  staff  briefings  (e.g.  mission  brief).  The  intention 
was  to  promote  a  high  level  of  acceptance  and  face  validity  of  the  IOPP. 

The  experimental  evaluation  represented  the  first  application  of  the  IOPP  in  a  simulated 
operational  context.  In  general,  the  results  suggest  that  teams  of  experienced  LF  personnel  were 
able  to  follow  the  IOPP  without  much  difficulty,  although  some  participants  were  somewhat 
reluctant  to  commit  to  a  single  COA,  likely  because  they  are  accustomed  to  developing  three 
COAs  (ideally)  with  the  OPP. 

5.2  Training 

A  web-based  training  course  for  both  the  OPP  and  IOPP  was  developed  to  allow  greater 
flexibility  in  the  delivery  of  the  training  than  lecture-based  training.  Aside  from  some  minor 
technological  complications,  delivery  of  the  training  courses  was  successful  and,  overall, 
participants  provided  positive  feedback  on  the  content  and  format  of  both  the  OPP  and  IOPP 
training.  They  did  suggest,  however,  that  the  web-based  training  should  be  supplemented  by 
some  form  of  face-to-face  training,  such  as  lecture-based  or  syndicate  group  work  training. 
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Participants  were  most  dissatisfied  with  the  lack  of  opportunity  to  ask  questions  at  any  time 
during  the  training.  This  would  be  addressed  by  incorporating  some  form  of  face-to-face 
interaction  into  the  training. 

Participant  feedback  suggests  that  incorporating  more  concrete  examples  of  how  to  apply  the 
IOPP  and  what  products  should  be  generated  would  have  made  the  training  more  effective.  In 
fact,  one  team  suggested  the  use  of  JPEG  or  MPEG  files  showing  specific  planning  products 
either  being  produced  or  completed.  They  also  indicated  that  additional  guidance  on  the  manner 
in  which  the  SA  step  is  carried  out  would  be  beneficial,  even  to  the  point  of  providing  a  bulleted 
checklist  of  SA  items  to  consider. 

5.3  Experimental  Evaluation  of  IOPP 

Overall,  the  results  of  the  experiment  provided  only  weak  evidence  that  the  IOPP  led  to  better 
planning  performance  than  the  OPP.  One  team  (Blue)  was  rated  consistently  higher  on  measures 
of  performance  when  using  the  IOPP  than  OPP.  This  result,  however,  must  be  interpreted  with 
caution  as  the  Blue  team  used  the  IOPP  on  the  second  day  of  the  exercise  and  likely  improved 
their  performance  simply  from  practice  with  the  planning  task.  Thus,  it  is  unclear  to  what 
extent,  if  any,  their  superior  performance  on  day  2  can  be  attributed  to  beneficial  effects  of  the 
IOPP.  The  other  team  (Yellow)  showed  virtually  no  differences  in  ratings  of  performance  when 
using  the  IOPP  and  OPP.  This  team  used  the  OPP  on  the  second  day  and,  if  a  practice  effect  did 
occur,  their  performance  when  using  the  OPP  may  have  been  boosted  relative  to  their 
performance  when  using  the  IOPP,  thereby  disguising  a  difference  between  the  two  processes. 
There  is,  however,  no  way  to  separate  the  effects  of  process  (IOPP  vs.  OPP)  from  potential 
practice  effects. 

Evidence  that  the  IOPP  works  better  as  a  process  is  also  tenuous.  Although  the  Blue  team  gave 
higher  ratings  on  all  measures  of  the  quality  of  the  process  and  products  generated  to  the  IOPP,  the 
Yellow  team  gave  essentially  equal  ratings  for  the  two  processes.  Again,  the  Yellow  team 
performed  the  OPP  on  the  second  day  and  their  perception  of  the  quality  of  the  OPP  as  a  process 
may  have  been  higher  as  a  result  of  practice  with  planning  in  general.  Nevertheless,  participants 
expressed  reservations  about  both  the  IOPP  and  OPP,  suggesting  that  they  saw  weaknesses  in  both. 

Both  teams  rated  the  usability  of  the  IOPP  higher  than  that  of  the  OPP.  The  IOPP  has  fewer 
steps  than  the  OPP  and  both  teams  were  able  to  complete  a  higher  proportion  of  those  steps.  The 
IOPP  also  seemed  to  facilitate  iteration,  or  looping,  of  steps  during  planning.  Despite  judging 
the  IOPP  as  more  usable,  participants’  ratings  did  not  indicate  any  difference  in  their  perceived 
workload  between  the  IOPP  and  OPP  conditions.  Rather,  the  main  factors  governing  perceived 
workload  were  the  session  day  (1st  or  2nd)  and  the  time  of  day  (morning  or  afternoon). 

Participants  also  generally  indicated  roughly  equal  levels  of  trust  in  the  IOPP  and  OPP,  but 
expressed  concern  that  the  IOPP  could  lead  to  more,  and  more  serious  errors. 

On  the  basis  of  this  experiment,  it  appears  that  the  IOPP  may  be  able  to  provide  benefits  to 
operational  planning  in  terms  of  both  efficiency  and  effectiveness.  This  potential,  however,  must 
be  further  examined  and  the  IOPP  revised  in  light  of  the  feedback  received  from  participants  and 
external  reviewers  of  the  process.  Subsequent  iterations  of  the  IOPP  will  address  participants’ 
concerns  regarding  the  potential  for  serious  errors  being  made  in  planning  and  consider  ways  to 
enhance  the  effectiveness  of  the  process  in  generating  creative  and  high-quality  planning  results. 
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It  must  be  noted  that,  although  generally  successful,  the  experiment  was  subject  to  several 
limitations.  The  participants  lacked  experience  with  the  OPP  and  had  also  never  worked  together 
as  a  team  at  the  battle  group  level.  Additionally,  given  the  context  of  the  experiment,  there  was 
a  certain  lack  of  realism  inherent  in  the  study.  A  battle  group,  for  example,  would  have  access 
to  more  resources  than  did  the  experimental  teams,  such  as  engineering,  artillery,  and  such.  In 
addition,  the  absence  of  outside  demands  allowed  the  Commander  to  be  more  heavily  involved  in 
the  planning  process  than  would  be  possible  in  an  operational  context.  Data  collection  was  also 
limited  by  the  fact  that  that  some  steps  in  both  the  IOPP  and  OPP  were  implicit  and  therefore 
difficult  to  identify.  This  was  especially  apparent  for  SA  steps  within  the  IOPP.  Difficulty  in 
identifying  steps  was  noted  in  previous  studies  (Bruyn  et  al.,  2005)  and  emphasizes  the  need  for 
observers  to  be  well  versed  in  the  planning  process  being  observed.  Finally,  there  was  an 
apparent  practice  effect  over  the  two-day  experiment,  with  participants  having  an  easier  time 
performing  the  planning  scenario  on  the  second  day  regardless  of  which  process  was  used. 
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6.  Future  Work 


There  is  considerable  opportunity  for  future  work  with  regards  to  the  IOPP  model,  training  of  the 
IOPP  and  future  experimental  evaluations. 

In  terms  of  the  IOPP  model,  further  development  of  the  model  should  be  pursued.  Publication 
and  communication  of  the  IOPP  to  a  broader  audience  is  essential  to  successful  development  of 
further  iterations  of  the  IOPP.  In  communicating  the  IOPP  to  a  broader  audience,  the  fact  that 
the  IOPP  incorporates  the  ‘best’  bits  of  existing  intuitive  models  and  incorporates  continuous  SA 
check  and  iterative  mission  analysis  and  COA  development,  should  be  emphasized. 

Future  research  should  specifically  address  criteria  for  success  and  SA  in  general  in  terms  of  how 
these  concepts  should  best  be  implemented  and  represented  in  the  planning  process. 

Several  opportunities  exist  for  training  enhancements  including  the  development  of  the  SA 
component  of  training,  improved  online  capabilities  of  the  training,  the  provision  of  24/7 
support,  online  conferencing  with  audio  and  visual  capabilities,  the  use  of  virtual  syndicates,  and 
the  use  of  more  concrete  examples. 

Future  experimental  evaluations  should  comprise  at  least  a  3 -day  exercise  in  order  to  circumvent 
first  day  familiarization.  Also,  the  involvement  of  more  planning  teams  would  be  desirable  in 
order  to  create  a  totally  balanced  experiment.  The  involvement  of  more  participants  would  also 
increase  the  validity  and  generalizability  of  results.  A  more  representative  training  environment, 
such  as  a  command  post  environment  in  a  field  setting  at  CFB  Borden,  Meaford,  Kingston  or 
Petawawa  may  also  enhance  the  realism  of  the  experience. 

In  terms  of  the  exercise  scenarios,  a  more  dynamic  scenario  that  better  represents  modem 
warfare,  such  as  asymmetric  warfare,  would  be  desirable.  Further,  if  the  experiment  calls  for 
two  scenarios,  the  scenarios  should  be  different  enough  that  there  is  no  learning  effect  between 
the  scenarios.  That  is,  planning  teams  should  have  to  do  an  equivalent  amount  of  mission 
planning  (especially  mission  analysis  and  building  SA)  for  both  scenarios.  With  respect  to  the 
presentation  of  scenario  information,  the  use  of  jpeg  files  and  the  provision  of  a  trace  rather  than 
a  paper  map  would  save  time  for  the  teams  and  allow  them  to  allocate  more  time  to  planning. 

The  team  would  also  likely  benefit  from  more  detailed  examples  of  expected  output  (e.g.  battle 
matrix). 
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8.  List  of  Acronyms 


AAR 

After  Action  Review 

AMPP 

Abbreviated  Military 
Planning  Process 

A00 

Area  of  Operations 

AOPSO 

Assistance  Operations 
Officer 

BARS 

Behaviourally  Anchored 
Rating  Scales 

Bde 

Brigade 

C2 

Command  and  Control 

CFS 

Criteria  for  Success 

Comd 

Commander 

CF 

Canadian  Forces 

CFB 

Canadian  Forces  Base 

CFC 

Canadian  Forces  College 

CFOPP 

Canadian  Forces 
Operations  Planning 
Process 

CIMIC 

Civil-Military  Cooperation 

CLFCSC 

Canadian  Land  Force 
Command  and  Staff 
College 

CMBG 

Canadian  Mechanized 
Brigade  Group 

COA 

Course  of  Action 

COO 

Concept  of  Operations 

COP 

Contingency  Plan 

COS 

Chief  of  Staff 

Div 

Division 

DRDC 

Defence  Research  and 
Development  Canada 

FOO 

Fire  Operations  Officer 

Frag  0 

Fragmentary  Operational 
Order 

HQ 

Headquarters 

Into 

Intelligence  Officer 

IOPP 

Intuitive  Operations 
Planning  Process 

ISTAR 

Intelligence;  Surveillance; 
Target  Acquisition; 
Reconnaissance 

JTF 

Joint  Task  Force 

LCol 

Lieutenant  Colonel 

LFCA 

Land  Forces  Central  Area 

LFWA 

Land  Forces  Western  Area 

LOGO 

Logistics  Officer 

MOE 

Measure  of  Effectiveness 

MOP 

Measure  of  Performance 

NDHQ 

National  Defence 
Headquarters 

ORBAT 

Order  of  Battle 

OPO 

Operations  Order 

OPP 

Operations  Planning 
Process 

Ops 

Operations 

OPSO 

Operations  Office 

PTA 

Primary  Training  Audience 

PUT 

Planning  Under  Time 
Pressure 

RA 

Research  Assistant 

RPM 

Recognitional  Planning 
Model 

SA 

Scientific  Authority 

SME 

Subject  Matter  Expert 

SOW 

Statement  of  Work 

TF 

Task  Force 
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Annex  A:  IOPP  Model  with  Training  Notes 


1 .0  Situational  Awareness 

Performed  continuously  by  all  personnel 
(Higher  Cmd;  Cdr;  Staff  and  Lower  Formations) 


Awareness  of 
Enemy 
Situation 


Awareness  of 
Own  Situation 


_ T _ 

Conduct  a 
meta-cogniti\£ 
check 

T 

4  A 

Awareness  of 

w 

Battlespace 

Environment 

2 

k 

Awareness  of 
Other  Military 
and  Qvilian 
Resources 


Situational  Awareness  Notes: 
-Own/Enemy 
Situation  -  resources, 
constraints,  opportunities 

-  Meta-cognitive  Check  (am  I 
working  in  line  with  Cdr 
intent?  mission?  is  this 
plan/COA  still  valid?  Also 
includes  noting  Assumptions 
and  Critical  Points  -  these 
should  be  noted  constantly  by 
the  Staff/Cdr  and  reviewed 
regularly  to  ensure  still  valid) 

-  Battlespace 

environment  -  geographical 
area,  political  situation,  climate, 
terrain,  cultural  awareness... etc; 

-  Resources  -  NGOs;  media, 
and  other  civilian  organizations 
that  share  our  battlespace  (and 
could  have  other  agenda's) 

-  Overall  this  process  is  meant 
to  display  that  not  everything  is 

a  discrete  action,  also  reminding 
personnel  to  ask  "what  is 
happening  around  ycu?",  "what 
does  it  mean?",  and  "how does 
it  relate  to/impact  other 
things?"...  similar  to  Common 
Operating  Picture  or  Situation 
Model  of  Battlespace 


Overall  Notes: 

-  the  size  of  all  the 
process  boxes  are  the 
same,  and  are  not 
meant  to  reflect  the 
ameunt  of  time  spent  on 
each  activity  (i.e..  some 
processes  take  a  great 
deal  longer  than 
others... ) 
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2.0  Mssion  Analysis 

Mssion 
Receipt  from 
HgherCmd 


Page  A-2 


Specific  Situational  Av\areness 
Aspects  to  focus 
on:  -  Assumptions  and  critical 
points  -  CWVEnemy 
Situation  -  Other  Military  and 
G\tlian  Resources 


Periamed 

Together 


LF  =  Lovver  Formations 


Mssion  Analysis  Notes: 
-Separate 'Review 
Objectives'  and  'Review 
Stuation'  as  2  distinct 
processes,  so  then 
when  'Review  Stuation', 
can  knew  how  that  fits  in 
with  objectives 

-  IW\briefing  -  indudes 
stating  Most  Ukely 
EOCA 

-  IPG  -  key  tirrings,  what 
known  of  mssion, 
addition  information 
required 

-Initial  Intent  -  desired 
end  state,  criteria  fa 
success,  desired  effects 

-  Overall  this  stage 
outlines  the  gathering 
and  analyzing  of 
information  speafic  to 
the  mssion 
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Staff  Commander 


3.0  COA  Development 


0) 


Sufficient 
/  info  to 
initiate  COA 
develop¬ 
ment? 


Yes 


Cdr  creates 
skeleton  COA  / 
Criteria  for 


No  Reinitiate 
►  Mission 
Analysis 


No 

Cdr  may  ask 
Staff  to  create 
skeleton  COA 
/  Criteria  for 
Success 
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Specific  Situational  Awareness 
Aspects  to  focus  on: 

-  Assumptions  and  Critical  Points 
-  Meta-cognitivs  Check 


Staff  Create 
Skeleton  COA 


No 

A 

Approvs 
Skeleton 
COA  / 
Criteria  for 
Success  ? 


Yes 


Staff  Refine/ 
Develop  COA 


COA  Development 
Notes: 

-  Staff  Refine  /  Develop 
COA  -  identify  taskings 
and  groupings,  C2 
structure,  develop  initial 
battlespace  graphics, 
initial  tactical  objectives; 
develop  COA  by  adding 
pieces  and  make  more 
robust.  Need  to  note 
assumptions,  critical 
points  and  identify  initial 
branches  and  sequels. 

-  Overall,  this  stage 
occurs  as  part  of 
Mission  Analysis  -  is 
separated  out  to  visually 
(in  the  diagram)  show 
the  growth  of  the  COA 
from  the  beginning 


Skeleton 
►  COA  given 
to  LF 
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COA  Validation  Notes: 

-  Staff  Estimate  is  the  same  process  outlined  in  the  current  OPP 

-  Viability  encompasses:  suitability,  feasibility,  acceptability,  and  completeness 

-  mental  Wargaming  (most  dangerous  ECOA,  branches  and  sequels,  assumptions,  critical  events,  decision  points, 
inconsistencies,  decide  whether  it  meets  criteria  for  success).  This  'mental  wargaming'  process  is  less  formal  and 
more  reflective  than  what  'wargaming'  currently  is.  Attention  must  be  paid  to  assumptions  and  critical  points  in 
mental  wargaming  (and  throughout  dev'mt/validation  process). 

-  during  Staff  and  Cdr  discuss  COA  -  work  out  final  issues,  agree  it  seems  viable  (at  end  of  approval  COA  developed 
as  a  CONOP) 

-  Overall  this  stage  involves  a  great  deal  of  questioning  of  the  viability  of  the  COA...  if  the  COA  is  not  validated  then  a 
new  COA  needs  to  be  started 
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Staff  Commander 


2  T3 


CONOP 

Approval 


5.0  Plan  Development 


Specific  Situational  Awareness 
Aspects  to  focus  on: 

-  Meta-Cognitivs  Check 

-  Own/Enemy  Situation 

-  Battles  pace  Environment 


Plan  Development 
Notes: 

-  intentionally  left  out 
create  Orders,  leaves 
open  the  choice  of  how 
to  deliver  Orders  (verbal, 
written,  etc) 

-  Plan  should  be 
expressed  in  terms  of 
initial  state,  desired  end 
state  and  transition  state 
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Annex  B:  Participant  Ratings  of  Online 
Training 


Question 

The  way  the  lessons  were 

PI 

P2 

P3 

P4 

P5 

P6 

P7 

Mean 

Min 

Max 

Std 

presented 

3 

2 

5 

5 

4 

4 

4 

3.9 

2 

5 

1.07 

The  use  of  example 

2 

2 

2 

4 

3 

4 

2 

2.7 

2 

4 

0.95 

The  use  of  graphics 

The  web-based  delivery  of 

4 

3 

4 

4 

4 

4 

3 

3.7 

3 

4 

0.49 

the  training 

The  convenience  of  the 

3 

4 

5 

5 

5 

2 

0 

3.4 

0 

5 

1.90 

training 

The  overall  effectiveness 
of  the  training  with  respect 
to  your  retention  of  what 
was  taught 

5 

4 

5 

5 

5 

4 

5 

4.7 

4 

5 

0.49 

2 

4 

4 

4 

4 

3 

3 

3.4 

2 

4 

0.79 

The  feedback  provided  by 
the  training  application 

2 

4 

4 

4 

3 

2 

1 

2.9 

1 

4 

1.21 

The  speed  of  the  system 

The  opportunity  to  ask 

5 

4 

4 

4 

3 

3 

2 

3.6 

2 

5 

0.98 

questions/get  support  from 
course  facilitators 

2 

2 

3 

3 

2 

1 

1 

2.0 

1 

3 

0.82 

The  information  imparted 
during  the  training 

3 

4 

4 

5 

4 

4 

4 

4.0 

3 

5 

0.58 

The  relevance  of  the 
training  to  your  role 

5 

4 

5 

4 

3 

5 

4 

4.3 

3 

5 

0.76 

The  value  of  the  training  to 
the  Canadian  Forces 

2 

4 

5 

4 

3 

4 

4 

3.7 

2 

5 

0.95 

Humanly  stems®  Incorporated 


Page  B-l 


This  page  intentionally  left  blank. 


Page  B-2 


Humans ystems®  Incorporated 


Annex  C:  Observer  Observations 


OPP  Blue  Team 

Time  Observations 

0:00  Comd  takes  ~5min  to  organize  team  and  give  initial  direction. 

0:01  Team  general  organization  of  materials 

0:11  Comd  gives  instruction  and  responsibilities 

0:11  Cmdr  checks  in  with  staff  to  see  if  they  have  the  necessary  tools 

0:11  Comd  says  to  start  initiation  process  -  think  about  factors  and  constraints 

0:11  10  to  focus  on  E  COAs  after  -  most  likely  most  dangerous  bridgeheads 

0:11  Comd  going  to  do  IPG 

0:12  Comd  goes  to  do  initial  planning  guidance  (physically  left  the  room) 

0:12  Cmdr  leaves  to  develop  planning 
0:14  10  investigates  enemy  positions 

10  is  reading  Intel  summary 

OPS  O  sorting  out  cheat  sheets  do  doest  forget  steps,  grouping  papers 
0:15  AOPS  O  putting  overlay  on  talc 
0:15  A  Ops  O  and  logistics 

0:15  10  referring  to  battle  staff  smartboard  for  maps,  symbols,  etc. 

0:15  10  doing  overlap  for  E  COAs 

0:16  Much  tracing  of  maps 

0:20  FOO  &  LOG  assess  ANNX  A  and  set  map 

Comd  still  developing  IPG  -  assumptions,  CCIRs,  comd  intent,  facts  and 
0:25  assumptions 

0:25  Looking  at  task  organization  matrix 

0:25  OPS  O  begins  to  look  at  overlay,  comparing  paper  to  talc 

0:29  Updating  of  maps  and  overlays 

0:32  Comd  requests  help  from  arty  (John)  later  in  day 

10  comes  to  get  enemy  doctrine  (how  they  attack,  breakdown,  order  of  march 
echelons,  how  they  deploy)  from  OPS  O 
0:34  In  a  real  war,  10  should  have  this  with  him 
0:36  LOG  an  AOPS  O  are  reviewing  big  map 
0:37  OPS  O  and  AOPS  O  ask  if  bridges  have  been  identified 
0:38  Comd  announces  he  will  deliver  IPG  in  5  min 
0:38  Comd  announces  that  initial  planning  guide  will  be  released 
0:38  OPS  O  and  AOPS  O  are  mapping  text  to  map 
0:39  Comd  observes  OPS  O  and  AOPS  O  doing  above 
0:40  OPS  O  returns  to  desk  and  interprets  more  info  from  package 
0:42  OPS  O  and  AOPS  O  interpreting  map  -  ambiguity 
Comd  tells  people  where  he  wants  them  to  sit 
OPS  P  and  AOPS  O  are  discussing 
Everyone  else  is  looking  over  notes 
0:42  Comd  says  'listen  up' 

Comd  gives  intent  (as  outlined  by  higher  Comd??)  verbally  (will  also  provide  it  on 

paper)  and  timeline 

Comd  shares  some  of  his  guidelines 

Passes  assumptions  onto  to  OPS  O:  specifically  what  we  have  to  do,  CIRs  to  think 
about 

10  needs  to  understand  enemy  speed 
0:42  Questions? 
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Time 


Observations 

Plan: 

10:30  need  mission  analysis  brief  COA 
13:00  mission  brief 
15:00  all  products  generated 
15:15  practice  presentation 
0:42  15:30  end 

0:42  Comd  tells  OPS  O  his  more  specific  needs  -  time,  speed,  distance  of  enemy 
Comd  delivers  IPG  -  asks  for  update  from  A  Ops  O  10  (all  present)  for  enemy 
update 

0:44  44-56  mins 

IPG  -  intent,  role  of  4RCR,  timings  (48hr),  mission,  COIR  -  tells  10  need  to  get  good 
0:44  idea  of  how  fast  enemy  can  move 
0:44  Timings  for  rest  of  day  -  to  be  posted 

0:44  To  present  msn  brief  @1030  and  debrief  1300,  products  generated  1500 

0:55  Cmdr  leads  team  meeting,  asks  for  assumptions 

0:56  OPSO  assigned  task  of  mission  statement 

0:57  FOO  assinged  task  of  "shaping  and  channelizing"  the  enemy 

LOG  assigned  task  of  managing  the  dumping  stations  for  combat  supplies,  and 
0:58  transfer  of  supplies 

1 :00  10  is  asked  to  prepare  a  time  statement  of  enemy  speed  and  volume 

1 :02  OPSO  gives  10  ORBAT  -  calculates  when  they  will  see  lead  elements 
1 :02  OPS  O  gives  10  ORBAT  ->  speed,  distance,  time 

LOG  looking  at  big  map  with  AOPS  O  ->  routes,  dump  sites,  movement  of 
1:03  humanitarian  aid,  ammunition,  rations 
1 :06  FOO  posts  timeoutline  on  door  reflecting  guidelines 
1 :08  Comd  meets  with  Ops  O  to  do  msn  analysis 

Comd  and  OPS  O:  clarification  and  brainstorming  (Comd  telling  OPS  O  what  he 
wants  said  in  the  Mission  brief  because  OPS  O  doesn't  know  much  about  Mission 
1:08  Analysis 

1:10  LOG  confers  with  AOPSO 
1:16  Done  first  map 

1:17  OPS  O  writing  out  Mission  Brief  using  Comd,  map,  and  Frag  material 
1:19  Comd  and  10  meet  - 10  needs  to  create  TALC 

1 :20  LOG  &  FOO  agree  on  map  tracing  and  strategy  re:  firelines  and  supplies 
OPS  O  and  AOPS  O  work  on  mission  brief  together 
1 :24  -  Using  a  'mission  analysis  proforma'  and  Comd's  assumptions 

1 :24  FOO  confers  with  Cmdr  to  check  enemy  positions 
1 :25  LOG  confers  with  Cmdr  on  barriers  and  logisitic  planning 
1 :25  LOG  confers  with  OPSO  re:  trace  overlay 
1 :26  FOO  convers  with  OPSO  on  the  Mission  Analysis  brief 
1 :28  Mike  sent  to  assist  10  in  developing  timeline 
1 :28  Comd  wants  timeline  that  is  rolling 
1 :28  10  explains  2Bde  3km  east  @  75%  effective neess 

1 :28  Templating  enemy  timeline  onto  talc 
1 :28  FOO  works  with  10  on  OPFOR  time  and  space  assumptions 

Cmdr  and  LOG  discuss  NSR,  combat  supplies,  Adm  company,  resources 
1 :32  necessary  to  secure  flank 

1 :34  Cmdr  gives  LOG  responsibility  of  situating  the  reserves  to  protect  the  North  Flank 
1 :36  *Note:  Kent  giving  assumptions  to  10  -  helps  them  out  with  relevant  factors 
1 :38  Cmdr  receives  input  from  SME 

OPS  O  asked  Comd  for  info,  was  told  to  ask  SME 
1:42  OPSO  talking  to  SME 
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1 :43  *Kent  helping  Ops  O  trying  to  figure  out  1 1  hour  -  look  @  timings  in  Frag  O 
1 :45  LOG  completes  trace  overlay 

1 :47  Comd  asks  if  going  to  meet  timing  for  min  analysis  brief  -  "No" 

1 :53  Changed  timings  of  msn  analysis  brief  onward 

1 :54  10  helper  (floater)  asks  for  Comd's  help  with  timeline  and  location  of  enemy 

1 :54  Comd  says  MA  brief  in  1 5min  (moving  it  in  1 5min  increments) 

1 :54  10  returns  with  trace/overlay 

1 :54  FOO  confers  with  Cmdr  re:  timing  of  guidelines  to  staff 
1:55  Cmdr  changes  Mission  Analysis  Brief  to  11:15AM 
2:00  Comd  says  want  to  spend  more  time  on  MA  b/c  important 
2:00  Ops  O  asks  Kent  for  more  info 

2:04  Cmdr  and  LOG  discuss  North  Flank  defense,  boundaries,  and  troop  deployment 
2:10  Picture  taken  of  enemy  location  on  10  overlay 

Comd  tells  OPS  O  that  he  should  call  the  meeting  when  he  is  ready.  Meeting  set 
2:10  for  11:20 

2:12  Mission  Analysis  brief  set  to  1 1 :20AM  by  OPSO 

2:15  All  attend  mission  brief  statement 

2:16  MA  -  all  with  Ops  O  leading 

2:16  Log  O  -  enemy  location  and  kill  zones  for  friendly 

2:16  Ops  O  -  Comd  intent,  implied  tasks,  resources  (too  low),  mission,  assumptions 
2:16  Comd  asks  to  repeat  enemy  assumptions 
2:16  Mission  Analysis  brief  begins 

2:17  OPSO  begins,  LOG  "christens  the  grounds"  and  explains  setup 
2:18  OPSO  sets  mission  and  its  implied  tasks 
2:19  Cmdr  works  with  assumptions 

2:20  Cmdr  redesigns  of  the  mission  analysis,  timing  assumption 
2:28  Comd  asks  to  repeat  msn  statement  -  helps  craft  mission  statement 
2:28  10  states  assumptions  for  enemy 

2:28  Comd  asks  how  long  till  advance  guard  into  kill  zone?  *Critical  timing 

2:28  Comd  gives  Ops  O  assumption  re  northern  flank  -  needed  for  COA  development 

2:28  OpsO-CCIR 

*Kent  acts  as  Bde  G3  -  gives  assumptions  -  won't  see  bridge  crossing  til  light 
2:28  tomorrow,  422er  will  withdraw 

2:28  Provided  critical  timings  so  that  they  can  move  planning  along 

2:28  *Kent  gives  hints  on  how  to  do  staff  analysis 

2:45 

2:50  Finish  msn  analysis  brief 
2:50  Finish  mission  analysis 
2:59  Lunch 

2:59  Break  for  lunch,  planned  30m,  lasted  1  hr 
3:00  Lunch  3:00:00  to  4:00:00 
3:56  NASA  TLX 

3:56  Return  from  lunch,  fill  in  NASA  TLX 

Comd  gives  CPG  review  -  COA  development,  gives  considerations  and  sets  the 
picture 

4:11  Tells  OPS  O:  2  Fr  COA  and  flank 

Comd  gives  direction  for  COA  development  -  2  friendly  COAs  and  E  most  likely  and 
4:12  most  dangerous 
4:12  Lists  out  7  considerations  for  COAs 
4:12  *Took  picture  of  COA  considerations 
4:12  Cmdr  uses  talc  to  set  up  battle  positions 

4:16  10  works  on  developing  OPFOR  COA  (most  likely,  most  dangerous) 
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4:18  OPSO  works  on  developing  2x  FF  COAs 
4:21  Comd  drawing  on  flipchart  -  picture  * 

4:21  Ops  O,  10  and  Mike  (floater)  mtg  @  chart 
4:21  10  reviews  possible  enemy  actions 

4:23  OPS  O  and  10  are  trying  to  figure  out  where  the  enemy  is  going 
4:23  OPSO  organize  FOO  &  10  to  discuss  enemy  COA  and  set  barriers 
4:30  AOPS  O  is  creating  matrix  that  will  be  passed  on  to  subordinates 
4:33  Comd  asks  10  to  map  most  likely  and  most  dangerous  E  COA  on  talc 
4:33  Mike  (floater)  working  on  Arty  Plan 
4:34  Cmdr  confers  with  LOG  &  FOO  to  discuss  artillery 

4:36  OPS  O  is  aware  of  most  likely  E  COA  so  can  start  to  think  of  own  COA  (troops,  fire) 
4:38  10  drawing  most  likely  E  COA 

4:38  Cmdr  meet  with  10  privately 

4:39  FOO  inspects  C0A1  in  response  to  OPFOR  most  likely  COA 
4:40  FOO  confirms  w  AOP 

4:42  Team  called  over  for  COA  development,  LOG  raises  question  of  firing  range 
OPS  O  and  Comd  work  on  COA  development 
4:42  OPS  O  and  LOG  also  working  together 
4:50  Ops  O  drawing  COAI  on  top  of  most  likely  E  COA  (another  talc) 

4:50  *only  doing  2  COAs  for  most  likely  E  COA  because  of  ex  time 
4:54  A  Ops  O  drawing  COA2  on  talc  (with  logistics) 

4:54  Ops  O  works  more  on  COAI  -  synchronization  and  phasing  of  fires 
4:54  COA  1  complete 

4:55  2nd  COA  development:  LOG,  Cmdr,  AOPSO,  and  10 

FOO  and  OPS  O  working  on  details  of  COA  1  (synchronization,  phasing,  fires, 
timing  of  fires,  type  of  fires) 

AOPS  O,  LOG,  10  working  on  COA  2 
OPS  O  and  FOO  working  on  COA  1  ->  measures  to  consider 
AOPS  O  and  LOG  working  on  COA  2  ->  measures  to  consider 
AOPS  O,  LOG  and  FOO  working  on  COA  2  ->  measures  to  consider 
4:56  OPS  O  working  on  COA  2  ->  measures  to  consider 
4:57  Cmdr  sends  FOO  to  work  with  OPSO  on  fireline  for  COAI 
4:59  FOO  suggests  timing  of  fire  to  OPSO  type 
5:04  10  working  on  most  dangerous  COA  on  talc  *took  picture 

5:04  LOG  &  AOPSO  work  on  COA2  -  pros/cons 

5:06  Non-traditional  but  LOG  is  helping  out  working  through  COA  considerations  set 

5:08  FOO  &  OPSO  works  on  COAI  artillery  to  shape 

5:09  A  Ops  O  and  log  O  drawing  COA2  on  top  of  most  likely  E  COA 

5:14  Comd  wants  Ops  O  to  present  COAI  in  5  min 

5:14  Comd  tells  OPS  O  to  brief  group  in  COA  1  in  5  mins 

5:17  FOO  updates  COA2,  team  meeting  COA  development  presentations 

5:18  Mike  (floater)  doing  fire  plan  on  COAI 

5:20  presents  COAI 

5:20  10  -  most  dangerous  and  most  likely  COA 

5:20  Info  brief! 

OPS  O  reports  to  entire  group 

- 10  presents  most  dangerous  and  most  likely  enemy  COA 

-  OPS  O  presents  Fr  COA  1 

-  AOPS  O  presents  friendly  COA  2 

Questions  and  future  direction  from  Comd  throughout.  What  does  Comd  want  for 
5:20  decision  brief 
5:21  Meeting  commence 
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5:25  Ops  O  presents  COAI 

5:25  Comd  gives  further  direction  on  how  to  develop  COAI 

5:25  Comd  asks  him  to  discuss  how  COAI  address  7  COA  considerations 

5:25  COAI :  10  and  OPSO 

5:30  COAI :  Cmdr  gives  ISTAR  focused  feedback 

5:31  Ops  O  presents  COA2 

5:31  *Note:  Comd  looking  @  OPP  process! 

5:31  Comd  provides  more  direction  -  need  more  detail  for  ISTAR,  etc 
5:34  COA2:  AOPSO 
5:36  Decision  brief  @  6:00  ex  time 
5:36  Set  decision  brief  for  6:00:00 
5:38  Informal  brief  led  by  Comd... would  like  wargaming 
5:41  AOPS  O  and  LOG  thinking  through  issues  raised  by  Comd 
5:42  LOG  and  AOPSO  consider  AA  for  COA2  synergy  fire 
5:42  Comd  gives  informal  instructions  for  wargaming 
Comd  gives  wargaming  outline  -  Formal 
- 10  (enemy)  and  OPS  O  (us)  ->  wargame  COA  1 
5:43  -  lo  (enemy)  and  AOPS  O  (us)  ->wargame  COA  2 

5:43  Wargaming  as  a  team 

5:44  Wargaming  - 10  plays  enemy;  Ops  O  plays  friendly  -  all  present 
5:50  Offer  a  COA3 

5:59  Comd  wants  Ops  O  to  recommend  COA  and  present  decision  brief  right  away 

5:59  Comd  asks  OPS  O  to  wrap  up  and  give  recommendation 

5:59  LOG  adds  input 

5:59  Cmdr  suggests  finalization 

6:02  OPS  O  suggests  modifying  COA  1 

6:03  Ops  O  decides  on  abbreviated  COAI  (A) 

6:03  Comd  says  decision  brief  @  6:10 
6:03  *doing  expedient  comparison  of  COAs 
Decide  on  COA  1A 

6:03  Decision  Brief  to  take  place  at  6:10:00 
6:04  Cmdr  selects  COAI  a 
6:05  AOPS  O  has  to  leave  for  the  day 
6:07  Cmdr  reassigns  Decision  Brief  to  6:10:00 
6:10  Working  on  last  details  of  COA  1A 
6:11  Ammend  COAIa 

6:13  Realize  that  they  haven't  done  matrix  comparison 
6:16  Doing  matrix  comparison 
6:16  OPS  O,  10  and  LOG  begin  matrix 
6:16  Decision  matrix  composed  for  COAI,  COA2,  COAIa 
Decision  Brief 

-  ECOA  presented  by  10 

-  OPS  O  and  LOG  present  two  COA  (COA  1 A  and  COA  2)  followed  by  preferred 
6:20  COA 

6:22  Decision  brief 
6:22  10  enemy  COAs 

6:22  Ops  O  presents  COAI  *adv  and  disadvantages  of  each 
6:22  Log  O  presents  COA2 

6:22  Ops  O  presents  COA  comparison  matrix  with  Comd's  COA  considerations/criteria 
6:23  Decision  brief  commence 

Matrix  presented  (both  COA  compared  against  COA  considerations  as  outlined  by 
6:34  Comd) 


Yhxmansy  stems®  Incorporated 


Page  C-5 


Time  Observations 

6:34  Compare  COA1  &  COA2  along  seven  considerations 
6:36  Comd  concludes.  Staff  is  recommending  COA  1 A 

6:38  Comd  wants  Ops  O  to  work  up  COAI  but  wants  answer  for  fire  support  issue 
Comd  gives  authorization  for  COA  1A.  Can  get  on  with  grouping  and  tasking 
6:38  matrix) 

6:38  COAI  is  authorized  by  commander 

6:39  Told  to  work  on  grouping  and  tasking  matrix 

6:39  Grouping  and  tasking  matrix  deadline  established  for  6:48:00 

6:42  OPS  O  and  LOG  begin  tasking  and  grouping  matrix 

6:52  Ops  O  and  A  Ops  O  preparing  taskings  matrix 

6:56  Comd  is  writing  mission  statement.  He  is  preparing  for  plan  presentation 

Conops  developed,  approved,  and  plan  developed  and  approved,  and  delivered  in 
6:57  presentation 

6:58  (****  all  events  occurred  nearly  simulatenously  ****) 

7:00  Presentation  of  orders! 

7:00  Comd  -  repeats  mission  and  mission  statement 
7:00  10  -  present  E  COA  -  most  likely  and  most  dangerous 

7:00  Comd  presents  COAI  modified 
7:00  Introduces  plan  for  main  effort 

7:00  Questions? 

7:00  Discuss  timings 

7:00  How  to  close  off  back  of  killing  zone 

7:00  Presentation  of  plan  (SME  acting  as  High  Comd) 

7:10  Done!! 

7:10  OPP  completed  for  Scenario  A,  Blue  team 
7:10  Discussion  (questions  to  SME) 

7:10  NASA  TLX 
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0:00 

commander  asks  group  to  take  15  minutes  to  read  in  their  roles 

0:02 

engineer  states  that  OPSO  is  different  from  what  was  explained  to  them 

0:03 

Comd  tells  group  to  take  15  min  to  read  over  scenario,  familiarize  with  role  (all  are  at 
desks  reading) 

0:05 

engineer  examines  back  map 

0:06 

eng  asks  for  actual  trace  of  map--commander  suggests  looking  in  folder 

0:07 

OPSO  is  highlighting  and  reading,  EngR  is  looking  at  maps  and  and  asking  Bob  about 
trace 

0:08 

all  reading  quietly 

0:10 

2  staff  members  examine  back  map  and  ask  whether  they  can  print  stuff;  they  discuss 
and  clarify  state  of  enemy  attack 

0:10 

OPSO  comparing  book  map  and  classroom  support  template,  then  comparing  book 
map  with  wall  map 

0:12 

OPSO  is  joined  by  AOPSO  at  WM,  adressing  where  objective  is  going,  KZ  locations, 
concerns  of  taskings 

0:14 

eng  examines  map 

0:17 

eng  examines  map 

0:17 

Bob  talking  with  Comd  about  what  they  are  trying  to  prevent,  BG  arrangment,  other 
info  in  package  then  going  to  map  together  and  discussing  locations 

0:18 

Hill  asks  FOO  info  on  one  of  the  sheets  in  the  folder 
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0:19 

eng  examines  map 

0:19 

OPSO  and  10  discussing  expectations  (10  asked  OPSO),  then  discussion  of 
constraints  as  going  through  package 

0:21 

Comd  looking  at  files  on  CD,  Orbat...  trying  to  determine  what  RCR  1 , 2,  3  is 

0:22 

eng  approaches  map;  FOO  joins  eng 

0:23 

AOPSO  joins  in  Bob's  presentation  with  Comd  about  what  is  in  the  BG 

0:25 

eng  tries  to  clarify  what  his  role  is.  FOO  helps  him  with  it. 

0:25 

Comd  and  OPSO  talking  at  WM,  outlining  timelines  to  get  to  skeleton  COA  (looking  at 
IOPP  chart),  trying  to  determine  how  to  capture  metacog 

0:27 

eng  asks  what  an  ATDM  is;  FOO  tries  to  help;  eng  asks  Bob 

0:27 

commander,  FOO  and  OPSO  discuss  IOPP  by  process  map;  Eng  joins  them 

0:28 

commander  gives  OPSO  task 

0:28 

Comd  rearranged  room 

Comd  recommended  15  minute  reading  in  about  situation 

0:28 

OPS  O/AOPS  O  met  at  map  1  for  1  min  and  then  returned  to  desks 

AOPS  O  queries  ATG  with  FOO.  OPS  O  got  involved.  ENG  got  involved 

0:28 

AOPS  O  goes  to  map  to  locate  items  noted  in  Operation  Order 

OPS  O  discusses  what  is  required,  want  Comd  IPG 

0:28 

Review  structure  of  4CMBG,  what's  available 

0:28 

OPS  O  discussing  IOPP  with  Comd  and  FOO 

0:28 

OPSO,  10,  FOO,  AOPSO  still  discussion  metacog 

0:29 

AOPS  O  puts  talc  on  map  #3 

0:29 

ENG  turns  up  to  listen  to  COMD 

0:31 

COMD  noted  that  SA,  awareness  of  entire  situation,  current  situation,  civil  etc.  won't 
change,  therefore  meta  cognitive  check  regularly  to  determine  if  what  is  being 
planned  still  makes  sense. 

0:31 

Comd  assigns  OPSO  to  figure  out  how  to  document  metacog,  and  ensure  that  it  gets 
covered 

0:32 

commander  asks  FOO  to  make  list  of  assets  down  to  platoon  and  det  levels 

0:32 

Comd  wants  blow  up  of  KZ  and  and  other  areas  from  10 

0:32 

Comd  talking  to  FOO  about  what  he's  doing 

0:33 

Comd  getting  information  needed 

33-35  mins 

0:38 

AOPS  O  and  ENG 

0:39 

eng  and  AOPSO(?)  map  out  battle  space;  they  have  trouble  tracing  from  photocopy 

0:40 

OPS  O  reviews  specific  items  in  meta  cog  check  (Tab's  laptop) 

0:40 

Bob  talking  with  FOO  about  Orbat 

0:40 

10  and  OPSO  discussion 

0:40 

Comd  asking  about  laptop  usage;  Comd  takes  a  laptop,  EngR  takes  other 

0:42 

ENG  started  laptop  on  Map  3  to  get  more  detailed  view  of  trace 

0:42 

Comd  started  laptop  on  counter  at  side  of  room 

0:44 

OPS  O  discussing  sync  matrix  with  10 

0:45 

FOO  posts  asset  list  on  side  bulletin 

0:45 

Comd  puts  up  ISTAR  matrix,  and  looks  over;  10  talks  with  OPSO  still 

0:49 

OPS  O  mentoring  10  in  what  to  do 

0:49 

Comd  discussing  with  Bob  about  squadrons,  who  is  mission  and  files  on  CD 

0:49 

10,  AOPSO,  and  EngR  took  look  at  WM 

0:51 

OPS  O  moved  flipchart  to  beside  map  #1 

0:54 

FOO  asks  commander  if  asset  list  is  sufficient 

0:54 

OPSO  outines  'rhythm'  of  flappaper 
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0:54 

10  reading  over  info,  noting  maps  of  interst 

0:54 

Bob  talking  with  AOPSO  about  what  is  in  North,  south,  west... 

0:54 

AOPSO  creates  overlays 

0:55 

OPS  O  defined  battle  rhythm  of  flipchart,  confirmed  with  Comd  and  will  brief  rest  of 
cell  at  10:00 

0:55 

Broken  down  to  deliverables  not  steps: 

10:00  Staff  meta  cog  check  and  every  hour  as  required  (1 .5) 

12:00  CPG  (2.9) 

12:30  Skeleton  COA  issued  (3.2) 

13:30  Validated  COA  issued  (4.8) 

14:30  Sync  War  Gaming  complete  (5.2) 

15:00  Orders  complete  (5.5) 

15:30  Brief  to  Cmd  (End  State) 

0:56 

eng  and  FOO  update  asset  list 

0:58 

Comd  directs  that  in  2  mins  will  brief 

0:58 

Comd  and  10  (EN  ECOA)  -  High  Cmd  asked  about  strength  of  En  3  Bde  at 

Gananoque  river 

0:58 

Comd  -  Assigned  and  implied  tasks.  CONOP? 

0:59 

OPSO  outlines  time-line  to  group 

1:00 

OPS  O  briefed  battle  rhythm 

AOPS  O  queried  actual  format  of  orders 

Detail  of  meta  cog  check 

Brief  was  arranged  quickly  and  informally 

1:00 

Went  around  staff  to  give  brief  update: 

10  -  ECOA  briefed:  intent  is  to  achieve  tactical  surprise 

FOO/support  arms  advisor,  need  to  develop  fire  plan 

Synch  matrix  H  --H  and  20:  only  has  a  bunch  of  tasks,  complex  set  of  tasks 

Comd  cut  in  to  focus  FOO  to  task  at  hand 

Comd  asked  AOPS  O  to  pay  attention  to  briefing 

OPS  O:  who  is  responsible  for  CAS 

COMD  keep  team  on  track.  Not  to  plan  too  far  forward 

ENG  briefed  what  can  be  used  in  KZ  D1  and  D2,  no  time  and  space  issues,  it  is  right 
there  ready  to  be  placed 

Comd  summarizes 

1:00 

AOPS  O  shows  on  map  how  Op  force  will  move,  how  own  force  will  react,  withdraw  to 
Infers  some  branches  and  sequels 

Probable  next  phases:  know  about  own  forces,  how  laid  out,  enemy  ORBAT  received 
Tracked  vehicles 

ENG  ->  implications 

1:00 

OPSO  leading  first  'formal'  brief,  all  attending 

1:00 

displays  battle  rhythm  (on  flapsheet),  and  'staff  co-ords'  (aka  metacognitive  checks) 
that  will  be  occuring  every  hour 

1:02 

metacognitive  check  update 

1:02 

each  member  of  teams  states  what  they  are  working  on 

1:02 

Comd  comments  on  what  each  member  is  doing,  also  emphasizes  the  role  of 
assumptions,  and  how  the  OPSO  is  documenting  them  throughout  the  process 

1:03 

FOO  gives  report  on  own  status;  FOO  asks  commander  for  time  window 

1:07 

eng  gives  status  report-makes  2  assumptions  and  1  deduction 

1:12 

Comd  clarifies  the  goals  of  phase  2  (phase  2  is  what  they  are  planning  for),  ensures 
that  all  members  have  a  shared  understanding 

1:12 

Comd  goes  through  SA  point  by  point  (5  stars  of  SA),  ensures  all  have  SA 
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1:12 

Comd  asks  Bob  about  issues  he's  not  clear  on... 

1:18 

someone  mentions  concerns  about  terrorists 

1:20 

AO  PS  O: 

Terrain 

Sensor  limitations 

Civilian,  NGO  etc.  -  not  much  known 

Terrorist  Organization  in  Gananoque 

Assume  no  refugee  corridors 

Comd  -  any  other  factors  that  anyone  can  suggest 

Any  helo  assets 

End  meeting  1:20 

1:20 

Another  trace  at  map  #1 

Comd  wants  another  collaborative  session  at  10:40 

1:21 

OPS  O  ->  map  #1 

AOPS  O  ->  map  #3 

1:22 

group  meeting  ends 

1:25 

10  talks  with  OPSO,  then  OPSO  back  to  wall  map,  10  back  to  desk 

1:25 

all  working  quietly 

1:30 

EngR  looking  for  App  2,  which  he  doesn't  have. . .  Bob  gives  him  a  brief  of  information 
that  he  needs 

1:30 

Others  working  quietly 

1:30 

OPSO  brings  Comd  over  for  discussion,  EngR  joins  in.  Discussion  of  'slow  go'  and  'no 
go'  areas  of  map 

1:30 

AOPSO  at  TM  adding  to  the  overlay 

1:34 

OPSO  at  wallmap  with  10,  discussion  HVU;  EngR  working  with  FOO 

1:37 

eng  consults  map 

1:37 

Comd  at  desk  working 

1:37 

AOPSO  at  TM,  filling  in  areas;  OPSO  at  WM 

1:39 

10  goes  over  to  TM  with  AOPSO 

1:40 

FOO  consults  map 

1:40 

OPS  O  and  10  at  map  #1  -  defining  ECOA 

AOPS  O  and  FOO  at  map  #3 

1:46 

OPSO  talking  to  Comd,  he  feels  they  are  at  Factors  and  Deductions,  at  1 1am  hoping 
to  have  that  complete  and  will  then  do  metocog;  Comd  wants  to  do  metacog  now  and 
then  at  11:10  wants  to  finalize  CPG) 

1:49 

FOO  consults  map  w/another  staff  member 

1:50 

group  meets  for  metacognitive  checks 

1:50 

Another  team  meeting.  Discussing  assigned  and  implied  tasks 

Have  moved  into  defining  ECOA 

1:50 

Formal  brief  at  TM  for  metacog:  Comd  working  on  taskings  and  objectives,  then  in  10 
mins  will  issue  IPG 

1:50 

Comd  outlines  own  tasks  in  CJTF,  clairfy  and  define  FIX 

1:50 

Comd  discusses  movement  through  area,  key  thing  the  Comd  wants  to  focus  on;  then 
goes  through  all  the  Implied  tasks  (asks  each  team  member  for  input  as  to  what  they 
each  think  the  implied  tasks  are),  discussion  about  how  to  work  with  other  groups  and 
if  they  can  change  their  boundaries 

1:50 

Comd  makes  sure  that  he  has  checked  in  with  everyone  before  moving  on 

1:58 

deduction  made 

2:02 

commander  asks  Bob  if  boundry  change  is  possible 

2:05 

OPSO  comes  up  with  other  implied  tasks  (Comd  directs  him  to  write  them  down) 
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2:12 

discuss  implied  tasks  (about  5  of  them) 

2:13 

Comd  says  1  figured  out  what  1  want  to  do  already 

OPS  0  recommended  breaking  to  give  Comd  time  to  develop  planning  guidance  by 
12:00 

Summarize  -  probably  have  developed  ECOA  but  need  to  formalize  it 

2:13 

Comd  talks  about  E  most  dangerous  COA 

2:16 

Meeting  ends 

2:17 

FOO  talking  with  10  about  how  fast  things  are 

2:17 

Bob  directing  EngR,  OPSO,  and  AOPSO  about  bge,  etc  (at  TM) 

2:17 

Comd  at  own  desk 

2:17 

OPSO  suggets  that  meeting  ends  so  that  commander  has  time  to  plan 

2:18 

meeting  ends 

2:19 

all  listening  to  Bob  except  Comd,  who  is  looking  at  IOPP  and  scenario 

2:20 

group  (with  out  commander)  meet  w/Bob  to  discuss  strategy 

2:22 

10  and  OPSO  at  wall  map  with  EngR;  playing  with  overlay  on  map  and  then  bringing  it 
over  to  table  map 

2:23 

Took  overlay  from  map  #1  and  put  it  on  map  #3 

2:26 

group  (with  out  commander)  maps  transparent  sheet  onto  another  one  (table) 

2:26 

AOPS  0  asked  what  the  OPS  0  wanted  him  to  do.  Wanted  him  to  help  10  define 
ECOA 

2:26 

OPS  0  checked  process 

2:27 

OPS  0  went  to  map  #3  to  discuss  ECOA  with  A  OPSO  and  10 

2:28 

FOO  and  EngR  discuss  metacheck...  FOO  says  more  behavioural  than  technical 

2:29 

eng  and  FOO  review  IOPP  chart  and  clarify  steps 

2:31 

FOO  reviews  map 

2:31 

FOO  and  eng  review  IOPP  process  chart 

2:32 

everyone  but  Comd  at  TM,  Comd  talking  with  Bob  about  how  to  outline  COA 

2:34 

group  (without  commander)  crowd  around  table  map  to  discuss  COA 

2:37 

10  drawing  on  TM 

2:38 

10  puts  ECOA  on  map  #3 

OPSO  is  listing  PIRs,  ECOA 

2:40 

OPS  0  writing  Draft  Mission  Statement 

2:41 

FOO  and  AOPSO  review  asset  list 

2:41 

OPSO  talking  with  Comd  at  Comd's  desk  on  responsibilities  and  information 

2:41 

10  talking  with  EngR  at  TM,  discuss  outlet  to  attack,  counter  attack 

2:43 

OPSO  joins  EngR  and  10  at  TM 

2:45 

Comd  with  Bob,  discussion  IOPP;  OPSO  waiting  to  talk  to  Comd 

2:45 

Eng,  Foo,  10,  Aopso  at  TM  talking  about  visibility  assessments 

2:49 

AOPSO,  FOO,  Eng  discuss  at  back  map,  then  move  on  to  asset  list 

2:52 

ECOA  discussion  involving  OPS  O/IO/ENG/AOPS  0/  FOO 

2:52 

FOO  and  AOPS  0  discussing  equipment 

2:53 

group  (without  commander)  meet  at  table  map 

2:53 

Comd  has  been  outlining  a  COA 

2:53 

10  and  Opso  at  TM,  drawing  things 

2:53 

Aopso  and  Eng  and  FOO  at  Eng  flap  paper  list 

2:55 

everyone  meets  to  present  most  likely  COA? 

2:55 

Comd  leads  "coord  brief"  around  table  map 

2:55 

Each  member  discusses  isses  (10  discusses  ML  ECOA,  Comd  says  that  is  also  MD, 
Comd  wants  10  to  start  thinking  about  control  measures,  etc  the  enemy  will  do) 

2:56 

ECOA  by  10 
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Comd  wants  more  development  on  ECOA 

2:58 

commander  suggests  that  10  modify  his  plans 

2:59 

Mission  statement  by  OPS  O 

2:59 

SA  check  on  mission  statement 

2:59 

Comd's  intent,  based  on  mission  statement  is  clarified 

2:59 

Mission  Statement  -  drafted  by  OPSO,  read  out 

2:59 

Comd  explicitly  goes  through  SA  issues  to  ensure  they  are  addressed  by  mission 
statement 

3:00 

discuss  mission  statement 

3:03 

test  whether  mission  statement  satisfies  purpose 

3:04 

assumption  made 

3:05 

Comd's  planning  guidance 

Establishing  what  is  known 

3:05 

AOPSO  outlines  roles  that  are  expected  (for  other  Blue  F) 

3:05 

Comd  went  through  list  of  different  applied  tasks  that  were  created 

3:05 

Comd  outlines  4  moving  parts  in  their  own  COA  (not  sure  about  groupings  and 
taskings  yet 

3:08 

commander  lays  out  skeleton  COA 

3:09 

Comd  approves  mission  statement 

3:10 

Time  is  constrained  therefore  Comd  provides  skeleton  COA 
-  3  phase:  deny  gap  (black  diamond  route)  to  force  enemy  into  D2  ;  fix  in  D2;  secure 
LoD  and  assist  in  counter  moves  through  D2 

Therefore  have  sufficient  info  but  time  is  constrained  and  so  skeleton  COA  is  provided 
Staff  to  go  away  and  figure  out  why  COA  won't  work 

3:12 

Comd  provides  outline  for  each  person 

3:12 

Comd  requests  feedback  later  on  about  whether  or  not  a  certain  aspect  is  a  good  idea 

3:14 

2  priority  tasks  assigned  by  commander 

3:15 

Additional  specific  planning  guidance 

Abandoning  KZ  D1  ->  why  is  this  a  good/bad  idea? 

How  do  we  'sense'  the  battlespace? 

3:16 

Comd  directs  all  to  stay  in  high  level  of  analysis,  and  punch  holes  in  what  has  been 
set  out 

3:21 

Briefing  ends 

3:26 

meeting  ends;  break  for  lunch 

3:26 

LUNCH! 

4:07 

OPS  O  at  map  #3 

4:08 

FOO  doing  visibilty  assessment  at  TM;  others  are  finishing  the  NASA  TLX 

4:12 

Whole  team  meeting  at  map  #3 

Comd  asked  for  update 

4:12 

Comd  discusses  what  they  are  all  doing;  FOO  bringing  up  concerns  to  Comd 

4:12 

10  is  templating  the  enemy 

4:12 

Comd  outlines  specific  info  that  he  wants 

4:13 

FOO  consults  map 

4:15 

OPS  O  and  AOPS  O  discussing  integration  with  other  units 

4:17 

ENG  and  Fires  Talking 

4:18 

4  meet  at  table  (including  commander)  to  discuss  COA 

4:20 

Used  Google  satellite  imagery  to  visualize  terrain  elevation 

4:22 

AOPS  O  locating  FOO 

4:28 

Full  group  discussion  on  view  ability  of  KZ  1  and  2  and  where  they  all  need  to  see 
(looking  at  gradients) 
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4:29 

AOPS  O  talking  to  FOO  at  map  #1  -  silos 

4:30 

OPS  O  still  at  map  #3  ->  briefing  COA 

4:33 

commander  asks  Bob  what  he  expects  for  outcome 

4:35 

FOO  and  AOPSO  meet  at  back  map  to  discuss  positioning  of  troops 

4:35 

Using  google  maps  to  get  layout  and  better  understand 

4:35 

Comd  clarifying  with  Bob  what  the  final  products  are 

4:35 

10  with  OPSO  at  TM 

4:36 

Comd  asked  for  progress  check 
-  Revised  time  for  validated  COA  to  13:55 

4:43 

Comd  Criteria  for  Success: 

-  No  enemy  move  past  rear  boundary 

-  No  enemy  on  South  of  diamond  route  west  of  BP  101 

-  LD1  must  remain  clear 

-  Retain  at  least  70%  combat  effectiveness 

-  Main  effort  =  BP  101 

Some  refinement  of  specific  issues 

4:43 

Final  approval  of  the  COA  (formal  presentation)  around  TM 

4:43 

Comd  presents  Criteria  for  Success  (he  acknowledges  this  should  have  been  done 
earlier) 

4:44 

eng  asks  OPSO  what  info  he'll  need  on  the  matrix 

4:44 

FOO  and  AOPSO  discuss  COA  strategy 

4:48 

Comd  encourages  everyone  to  talk  about  the  3  phases  of  the  COA 

4:49 

Lost  about  10  mins  on  clock 

4:49 

commander  gives  criteria  for  success 

4:51 

10  briefing  on  ECOA 

OPS  O  briefing  on  COA 

4:54 

staff  discuss  COA  +  criteria  for  success 

4:56 

10  gives  his  view,  then  OPSO  (seemed  to  be  some  major  breakthroughs  about  where 
to  locate  staff) 

5:01 

each  individual  explains  his  part  of  COA 

5:02 

ENG  briefing  on  COA 

5:04 

FOO  briefing  on  COA 

5:04 

FOO  talked 

5:11 

Eng  talked 

5:17 

Comd:  start  thinking  about  synch  matrix,  wargaming  at  14:45 

5:17 

Comd  outlines  what  he  wants:  OPSO  to  do  groupings  and  taskings,  Mortars  to  be 
finalized,  ENG  to  confirm  things...  back  to  discuss  in  15  mins 

5:20 

meeting  ends;  individuals  go  off  to  revise  COA 

5:20 

OPS  O,  AOPS  O  and  10  working  at  laptop  in  front  left  corner  of  room 

5:21 

all  working  independently 

5:27 

FOO  at  TM,  Eng  at  desk,  AOPSO  &  OPSO  &  10  discussing;  Comd  looking  at  his  wall, 
and  thinking 

5:31 

Comd  at  TM,  others  along  at  desks 

5:35 

Cds  calls  them  back  together 

5:35 

OPSO  led  through  plans,  10  led  thorugh  plans,  10  led  through  what  to  expect  from 
enemy 

5:36 

Wargaming 

5:36 

Comd  very  aware  of  time 

5:44 

Phase  2  of  their  actions 

5:49 

group  meets  to  discuss  COA  and  do  war  gaming 
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5:53 

Comd  gave  further  direction 
-  Staff  responsibilities  to  complete  planning 

5:54 

They  move  to  CONOP  (which  is  quickly  approved) 

5:55 

Plan/Order  creation  -  everyone  breaks  into  own  groups  (AOPSO  working  with  OPSO 
at  laptop) 

5:57 

Meeting  adjourned 

COA  validated 

CONOP  was  developed  with  all  previous  steps 
-  Assume  High  Cmd  has  been  briefed  and  CONOP  approved 

5:58 

Plan  being  developed 

5:58 

Orders  being  written 

5:58 

Contingencies  being  done  (by  10) 

5:58 

Comd  wrote  responsibilities: 

G2:  Terrain,  weather,  enemy  intent,  enemy  COA,  BIG  PIRs 

Comd:  Mission  statement,  criteria  for  success,  concepts  of  ops 

G3:  GP  and  Tasks 

Fires:  Fire  plan  and  CFSP 

ENG:  Barrier  plan 

6:06 

group  meets  again--commander  assigns  tasks 

6:07 

All  still  working  on  own 

6:10 

AOPS  O  assisting  10 

6:10 

ENG  talking  to  Comd  about  obstacles  and  concerns 

6:16 

10  at  TM,  Aopso  and  Opso  at  laptop  talking  with  Comd,  Eng  and  FOO  say  they  are 
pretty  much  done 

6:17 

Eng  and  10  discuss  at  TM 

6:31 

eng  approaches  table  to  talk  to  10 

6:33 

Back  brief  to  Comd  begins 

6:37 

Taped  presentation  begins 

6:42 

G3/OPS  O  -  Group  and  tasks 

6:46 

Fires  -  Artillery,  CAS,  FOO,  etc. 

6:49 

group  presents  to  Bob 

6:51 

ENG  -  creating  and  exploiting  obstacles  -  Barrier  Plan 

6:56 

Comd  wrap  up 

-  war  gamed  against  ECOA  -  most  likely  and  most  dangerous 

-  recommendations  to  support  subsequent  phases 

-  moving  lines  of  departure 

-  moving  rear  boundary 

-  requesting  all  CAS  sorties 

High  Cmd  Question  -  how  long  will  each  phase  take? 

7:09 

Finish  presentation  (which  is  16:25  real  time) 

7:23 

group  fills  out  NASA-TLX  and  MOPS 

OPP  Yellow  Team 

Time  Observation 

0:00  Reviewed  structure  of  armoured  Bdge 

0:00  Team  members:  Comd,  OPSO,  FOO,  EngR,  10  (only  5  of  them!) 
0:00  Staff  break  from  scenario  presenation,  all  start  reading 
0:00  Return  from  debriefing,  Cmdr  initiate  OPSO  to  organize  FOO  and  10 
0:03  Step  1  -  initiation 
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Fires  and  ENG  working  together  to  determine  structure  and  composition  (ORBAT)  of 
0:06  the  armored  BG 
0:06  Putting  info  on  flip  chart  1 
0:07  Cmdr  leaves  to  go  planning 
0:09  10  at  table  map,  taking  notes 

0:09  OPSO  looking  at  OPP 

0:09  FOO/Eng  taking  notes  off  weapons  listings  from  file  on  computer 
0:12  Cmdr  returns  with  a  purple  booklet  and  butcher  paper 
0:14  10  talking  with  OPSO  at  TM 

0:14  OPSO  is  looking  at  OPP  to  determine  timeline  for  deliverables 
0:14  Prepares  staff  guidance 

0:17  Comd  clarifying  the  number  of  troops  and  planning  forCPG 

0:17  OPSO  presents  a  timeline  to  the  Comd  of  how  to  follow  the  process  -  agreed  upon 

0:17  Eng  and  FOO  still  noting  wpns  from  computer  file 

0:18  Confers  with  SME  regarding  available  units 

0:19  Confers  with  OPSO,  agree  2minutes  until  issue  to  staff 

Comd  briefs  expectations  (see  COMD's  initial  guidance  below) 

0:20  Will  issue  Cornd's  initial  guidance  at  9:25 
0:20  OPSO  calls  staff  in  for  CPG 
0:20  All  attend 

0:20  Cmdr  continues  to  prep  staff  guidelines 

0:21  OPSO  organize  staff  at  Cmdr  informal  order 

0:23  Guidelines  to  staff  are  issued 

0:24  OPSO  to  split  up  tasks  among  team 

0:25  Decide  that  IPG  will  be  at  9:25 

0:25  Cmdr  working  on  initial  guide  briefing 

0:25  Cmdr  prepares  IPG 

0:26  Staff  return  to  what  they  were  doing 

0:26  OPSO  talking  with  Bob  about  when  H  hour  is 

0:26  Cmdr  confers  with  FOO  regarding  troop  composition 

0:26  Cmdr  checks  with  OPP  sheet  and  SME  to  confirm  objective  for  15:00 

0:27  Cmdr  reflects  on  H  hour  timing 

0:28  OPSO  at  laptop,  Eng  and  FOO  still  copying  info  from  computer  (writing  on  flapsheets) 
0:28  Begins  rough  draft  of  timeline,  taking  into  consideration  the  objective  deliverables 

OPSO  and  10  talking  at  TM  about  limits,  enemy  concentation,  etc;  leading  10  through 
0:32  what  factors  he  would  like  to  consider 
0:32  Comd  at  desk,  looking  at  map 

Exhibits  very  close  concern  for  adherence  to  the  process  and  makes  process  vs.  time 
0:33  considerations 

0:35  Fires  and  ENG  discussing  no  fire  line  etc.  at  map  #1 

0:35  FOO  with  Eng,  looking  at  where  everyone  is  and  where  wpns  are,  etc 

0:35  10  still  at  TM  taking  notes 

0:35  OPSO  at  his  desk 

They  are  starting  to  look  at  issues  that  they  need  to  deal  with,  and  what  they  need  from 
external  groups,  discussion  of  ML  ECOA  at  WM...  realize  that  they  have  to  follow  the 
0:35  process  though 

0:36  Begins  writing  the  IPG  and  develops  a  timeline 
0:39  OPSO  woking  with  Comd 

0:41  Confer  with  OPSO  regarding  faults  or  weaknesses  of  IPG 
0:41  Agree  that  the  IPG  team  brief  will  commence  in  2  mins 
0:42  OPSO  passes  msg  onto  team 

0:42  Cmdr  further  develops  IPG  as  on  overlays,  disc  with  FOO 
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0:43  Posted  own  ORBAT  on  far  wall 
0:44  Announce  IPG  in  one  minute 
Comd  initial  guidance 

2  x  En  COA:  Most  likely,  most  dangerous 

3  x  Fr  COA:  All  must  be  significantly  different 
Guidance  to  staff: 

1)  internal  coordination 

2)  external  coordination 

-  4  PPCLI 

-  4  RCR 

3)  Sp/Sub  Units 

-  4  PPCLI:  plan  and  current  location  start 

-  4  R22eR:  old  barrier  plan 

0:45  -  current  gun/AFV  state 

0:45  IPG  commence,  indicates  process  steps  reference  to 

IPG  -  schedule  for  planning  on  flapsheet;  scheduled  Mission  Analysis  Brief  (MAB)  at 
0:46  10:20 

0:47  Confer  with  OPSO 

0:48  Bob  gave  bit  of  Inti  brief 

Initial  Comd  Guidance 
930  Issue  ICG 
1020  Mission  Analysis  Brief 
1050  Issue  CPG 
1200  Lunch 

1240  Deliver  info  brief  on  En  and  Fr  COA  (2  &3) 

1330  Deliver  decision  brief 
1400  Gain  CONOP  approval 
1 500  Back  brief 
0:48  HR  not  before  1700 

ICG  2  -  In  Auth  -  4  PPCLI,  4  RCR;  Recce  -  RGPS  auth,  move  to  apple  pie  now;  MOV  - 
0:48  not  before  1 530 

0:49 

Meeting  begins  around  map  #3  (10,  OPS  O  Fires,  ENG) 

0:50  Mission  Analysis  -  implied  and  assigned  tasks 
0:50  Factors  and  deductions 

0:50  Team  confer  with  SME 

0:51  All  grouped  at  TM,  Bob  is  giving  info  about  other  groups  and  scenario 
0:51  OPSO  leading  group  in  what  needs  to  happen  before  MAB 
0:51  Comd  is  not  at  TM,  is  looking  at  OPP 

0:54  All  looking  at  TM,  discussion  how  to  create  Mission  Statement  (MS) 

0:54  OPSO  writing  down  pts  for  MS 
0:54  Looking  for  planned  and  implied  tasks 

1 :00  All  but  Comd  looking  at  ECOA  and  attack  plan,  confirming  goals 
1 :00  Cmdr  checks  map,  plans  to  check  with  team  during  development  of  mission  statement 
1 :05  OPSO  trying  to  confirm  MS 

1:11  Consulting  purple  book,  determining  what  other  aspects  need  to  be  included  in  MS 
1:15  Cmdr  develop  a  list  of  issues  and  questions 
Some  discussion  of  what  will  happen  after 
COG  =  C2 
Combat  power 

1:19  What  other  units  will  be  doing 

1 :22  All  still  at  TM,  looking  at  COA  possibilities  (2  or  3)  and  constraints,  factors 
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1 :22  Looking  at  list  in  CFOPP  for  MAB  format 

1 :22  OPSO  brought  back  on  topic,  say  that  not  at  COA  development  yet 
1 :27  Cmdr  confer  with  SME  regarding  primary  and  secondary  objective 
1 :29  Key  strengths,  decision  points  (from  CFOPP  book) 

Mission  analysis  meeting  finished 
1 :30  All  went  back  to  prepare  for  mission  analysis  brief 
1 :30  confirm  readiness  of  OPSO  to  commence  Mission  Analysis  Brief  at  10:20 
1 :32  Break  from  TM  to  do  final  prep  for  MAB 
1 :32  10  stays  at  TM,  taking  notes  and  writing  on  overlay 

1:32  OPSO  at  own  desk 

1 :35  Mission  Analysis  Brief  begins  around  map  #3 
1 :35  MAB  started.  Staff  present  to  the  Comd 
1:35  10  discuss  maps,  lead  elements,  limitations 

1 :35  Commence  Mission  Analysis  briefing,  initiated  by  OPSO 
1:36  10 

1 :38  OPS  O:  gave  mission  statement 

OPSO  gives  mission  analysis  (goals,  tasks),  criteria  for  success,  constraints,  CoG  for 
1:38  enemy  and  MS 

1 :39  On  going  Q&A  throughout  for  clarification  when  needed 
1 :43  Cmdr  clarifies  mission  objectives,  c  of  g,  location  of  OPFOR 
1 :44  Comd  then  clarifies  issues  that  he's  figured  out  (liked  the  MAB) 

1 :47  Comd  is  providing  additional  guidance 

1:50  Finalizing  mission  statement 

All  still  at  meeting,  Comd  outlining  that  he  will  do  the  CPG  at  10:50;  tells  staff  what  he 
1 :50  wants  them  to  do  in  the  mean  time 
1:50  Refines  mission  statement 
1:51  Finalized  mission  statement 

1 :51  Cmdr  states  1 0:50  for  CPG 

1 :52  Asks  for  train  analysis,  routs,  assigns  K2,  shaping  ground 

1 :53  Mission  analysis  brief  ended 

1 :53  Fires  and  ENG  worked  separately 

1:55  end  of  meeting 

1 :56  Begins  CPG  preparation 

1 :59  Cmdr  readies  CPG 

2:00  Comd  is  at  the  flapsheet,  creating  the  CPG 

2:00  Eng  and  FOO  at  TM,  writing  on  overlay 

2:00  OPSO  at  laptop,  10  is  helping  OPSO 

2:02  Cmdr  confers  with  SME  on  troop  and  unit  availability 

2:09  Bob  with  Eng  going  through  ORBAT;  others  working  independently 

2:09  CPG  commence,  Cmdr »  OPSO  »  Team 

2:10  CPG 

2:10  Comd  CPG.  Comd  outlines  mission,  intent... 

2:13  Mission  statement  confirmed 

2:21  Comd  looking  at  OPP  with  group  to  clarifiy  their  next  steps 

Comd  outlines  what  he  wants:  3  COAs,  2  ECOA  (ML  and  MD);  also  outlines  specific 
2:21  things  to  look  for  when  developing  the  COAs 
2:22  CPG  ends 

2:22  4  staff  gathered  around  bird  table 

2:22  end  of  meeting;  everyone  breaks  to  own  work  stations 

2:22  Cmdr  brings  staff  up  to  speed  on  OPP  map 

2:27  Finish  CPG 

2:30  Staff  working  together  at  TM;  Comd  on  flapsheet,  outlining  symbols 
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2:33  Comd  enters  discussion,  telling  what  he  wants  from  the  individuals 
2:34  Some  additional  Comd  analysis  guidance 
2:35  Cmdr  provides  guidance  on  attack  positions 
2:45  All  around  TM,  discussing  weapons  and  COA's  (3) 

2:51  Looking  at  copies  of  maps,  each  developing  1  COA  onto  a  map  (10,  Eng,  FOO) 

2:54  OPSO  talking  about  schedule  to  Comd  (lunch?);  supposed  to  do  info  brief  at  1 2:40 

3:01  OPSO  assisting  with  COA  development 

3:02  Cmdr  &  team  use/develop  new  strategy  for  creating  COA's 

3:03  Lunch 

3:03  LUNCH 

3:03  Lunch  begins 

3:43  Back  from  lunch,  filling  out  NASA  TLX 

FOO  and  Eng  at  TM;  OPSO  at  desk  with  OPP  book;  Comd  at  desk,  looking  at  flap 
3:47  papers  (schedule  and  IPG  info) 

3:55  Comd  looking  at  map  and  thinking 

3:59  Cmdr  investigates  and  thinks  about  COA  on  map 

4:00  Info  brief  on  COAs  around  bird  table 

Comd  queries  COAs  to  improve  solutions 
Get  rid  of  Fr  COA  3 
Agrees  with  En  COA  Dang/Likely 
Develop  Fr  COA  1 
Suggestions  for  improvement 
4:00  Develop  Fr  COA  2 

4:00  Proposed  another  COA  refinement  (almost  a  new  one) 

4:00  Info  Brief:  from  staff  to  Comd 

4:00  OPSO  led  through  issues  that  need  to  be  covered 

4:00  10  doing  Enemy  ML  and  MD 

4:00  Led  Comd  through  COAS 

4:00  Info  brief  begins 

4:07  Comd  asking  questions 

4:07  Cmdr  offers  advice  and  suggestion,  clarification,  and  considers  combat  power 

4:17  COA  refinement  and  development  along  with  Cmdr  guidance 

4:18  Comd  removes  a  COA  (so  only  2  now);  agrees  with  MD  and  ML  ECOA 

4:18  Wants  the  other  2  further  developed 

4:21 

4:26  Info  brief  ends,  staff  begin  to  refine  COA 

4:26  End  of  CPG 

4:26  Comd  goes  to  his  own  WM 

4:26  Staff  stays  around  TM,  discussion  what  the  Comd  suggested 
4:26  Info  brief  ends,  Cmdr  investigates  map 
4:34  Planning  for  decision  brief  (DB),  all  working  at  TM 
4:34  Comd  talked  with  staff  about  symbols 
4:40  Cmdr  further  develops  COA  maneuvers 
4:50  Staff  still  at  TM,  Comd  at  own  desk  looking  over 
5:01  Cmdr  begins  phase  *  unit  orders  Matrix 
5:13  Cmdr  confer  with  OPSO  to  ready  the  team  for  decision  brief 
Decision  Brief 

OPS  O  presented  pros  and  cons  of  each  Fr  COA 
5:14  Therefore  comparison  and  wargaming  has  been  done 

Recommended  COA  1  due  to  simplicity  and  speed  (15  mins  faster) 

70-75%  Combat  power  in  COA  1 
5:14  60-65%  Combat  power  in  COA  2 
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Already  have  eyes  in  road  in  COA  1 
Comd  prefers  COA  2 

-  PPCLI  move  at  same  time  -  need  to  give  them  space 

-  Draws  attention  away  from  PPCLI 

5:14  -  Addresses  EN  most  Dang 

5:15  DB 

Staff  present  COA  1  and  2  (positives  and  negatives  for  both).  Led  by  OPSO. 

5:15  Recommend  COA  1. 

Comd  asks  questions  (specifically  about  how  the  wargaming  went).  Likes  2  better. 

5:20  Comd  explains  why. 

5:20  Cmdr  disagrees  with  staff  selected  COA,  COA1  prefers  COA2 
Want  to  develop  COA  2  further 
5:23  Moving  into  plan  development  (Comd  said  so) 

5:23  Comd  goes  to  Bob  for  CONOP  approval 
5:23  Further  refines  COA2 
5:25  CONOP  approved 
5:25  OPSO  looking  at  OPP 

5:25  See  what  they  need  to  do:  identify  issues  and  shortfalls 

Comd  outlines  the  format  that  he  wants  for  issues  and  shortfalls;  But  then  Bob  says  that 
5:25  they  don't  need  to  issues  and  shortfalls.... 

5:25  Gains  CONOPS  approval 

5:27  Plan  Preparation,  Comd  involved 

Fires  and  ENG  working  separately.  Fires  writing  and  ENG  drawing  scheme  of 
5:27  maneuver 

5:29  Cmdr  works  with  Staff  on  CONOPS  development 

5:33  OPSO  writing  on  matrix 

5:33  Comd  writing  on  another  matrix 

5:33  10  and  FOO  at  TM,  creating  ovrelays 

5:33  Eng  at  another  flap  sheet 

5:35  Cmdr  filling  in  Orders  Matrix 

5:45 

5:55  Plans  done 

5:57  Start  back  brief  to  Bob 

5:58  Plan  back  brief  -  Comd 

5:59  Cmdr  gives  Issue  Plans  (back  brief) 

6:00  Team  involved  in  components  of  positions 

6:01  OPSO 

6:04  ENG 

6:06  Fires 

6:15  End 


IOPP  Blue 

Time  Observation 

0:00  Comd  gave  initial  guidance  before  scenario  briefing 

After  Lora's  presentation  group  took  5  minutes  to  discuss 
0:00  Bob's  situation  briefing 

0:00  commander  and  Logi  clarify  process  with  kent  at  butcher  paper  pad  (break-up  at  8:43) 
0:00  FOO  reads  notes 
0:02  Logi  leaves  room 

Comd  and  10  meeting  in  separate  room  discussing  enemy  -  identify  most  likely  place  to 
0:02  be  -  only  creating  most  likely  COA 
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0:02  945  for  MA  Brief  so  have  lots  of  time  to  do  enemy  situation 

0:03  OPS  O  and  AOPS  O  are  reading  Frag  O  to  try  and  understand  the  resources  they  have 
0:04  Foo  asks  for  booklet  that  contains  explanations  of  symbols 
0:09  Logi  returns  to  room 

0:1 1  Comd,  OPS  O  and  AOPS  O  are  discussing  what  they  can  do  while  looking  at  overlay 

0:1 1  Logi  and  FOO  each  going  over  information  from  folder 

0:13  FOO  asks  10  for  clarification  about  map  features 

0:15  Comd  and  A  Ops  O  looking  @  map  discussing  terrain,  etc. 

0:15  Discussing  applied  tasks 

0:15  Note:  Kent  tells  them  about  crossings  available/not 

Comd  does  TO&E  on  flipchart  to  confirm  what  assets  he  has 
0:15  8:42  (real  time) 

0:16  Logi  approaches  and  examines  map 

0:16  AOPS  O  returns  to  desk 

0:18  OPS  O  is  working  on  Implied  task 

0:18  OPS  O:  formal  Mission  Analysis  uses  sheet  as  an  outline  (same  as  yesterday) 

0:19  AOPSO  joins  Logi  at  map  and  both  leave  room 
0:21  Logi  returns  to  room;continues  examining  map 
0:21  Comd  reviewing  orders 

0:22  AOPS  O  states  a  possible  assumption  which  leads  to  discussion 
0:22  FOO  asks  10  for  more  clarification  about  typography  of  land 
0:23  Logi  returns  to  table;  starts  making  notes 
0:23  Comd  tells  OPS  O  and  AOPS  O  to  work  independently 

Staff  and  Comd  are  doing  individual  review.  Comd  is  sitting  and  reviewing  a  lot  more 
0:23  material  today 

Comd  periodically  talks  to  staff  when  he  notices  that  something  may  be  of  value  to  an 
0:23  individual 

During  mission  analysis: 

-  OPS  O  is  working  on  implied  tasks 

-  AOPS  O  is  working  on  facts,  deductions  and  assumptions 
Occasional  chatter  between  OPS  O  and  AOPS  O 

0:23  Communication  between  AOPS  O  and  OPS  O  becomes  more  frequent 
0:23  Aiming  for  mission  analysis  brief  (9:45  real  time) 

0:27  Logi  examines  map 
0:29  Logi  returns  to  seat 

0:33  Logi  approaches  FOO  to  discuss  enemy  situation  and  own  resources 
0:35  Comd  noting  tasks,  implied  tasks  and  factors 

Comd  asks  Ops  O  and  A  Ops  O  to  be  ready  to  discuss  factors,  deductions  and  tasks  @ 
0:35  930 

0:38  FOO  joins  commander  and  10  to  ask  a  question  about  enemy  positioning 
0:40  Comd  and  Log  O  @  map  talking  about  number  of  sortie's  to  use  -  decided  on  6 
0:40  Logi  joins  commander  at  map-they  start  discussing;Kent  joins  them  shortly  after 
0:42  Logi  goes  over  to  FOO  and  they  discuss 

*Comd  talking  to  Kent  about  most  likely  E  COA  (Comd  asked  for  his  input)  and 
0:45  assumptions 

Note:  Comd  asked  Kevin  (Log  O)  to  ensure  doing  situation  awareness  throughout  and 
0:45  ensure  they  provide  all  comms  to  lower  forms,  also  told  Ops  O  not  to  look  @  map  today! 
0:46  FOO  aks  10  for  clarification 
0:46  Logi  starts  to  make  diagrams  on  butcher  pad 
0:49  FOO  examines  map  on  wall 
0:51  FOO  asks  for  photocopy  of  his  diagrammed  map 
0:52  Comd  sits  down  with  OPS  O  and  AOPS  O  for  an  informal  briefing 
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Comd  asks  group  if  9:45  is  still  reachable,  mentions  that  can  move  it  to  10:00  (Comd 
0:52  showing  greater  flexibility) 

0:52  This  informal  briefing  is  more  discussion  and  analysis  oriented 
0:52  FOO  discusses  enemy  situation  and  own  assets  with  /IO 

Comd,  Ops  O  and  A  Ops  O  meet  to  discuss  assumptions,  deductions,  implied  tasks  and 
0:55  facts  -  informal  brief  to  allow  Comd  to  develop  skeleton  COA 
commander  asks  FOO  how  he  is  doing 
0:57  FOO  moves  to  a  seat  closer  to  the  rest  of  the  group 
0:58  FOO  asks  commander  question-Kent  helps  answer  question 
0:59  commander  asks  Logi  question 
1 :01  FOO  moves  back  to  orginial  seat 

State  assumptions  in  informal  brief  (Comd  wanted  this  informal  brief  so  he  could 
1 :03  visualize  skeleton) 

1 :03  Logi  joins  commander,  OPSO  and  AOPSO 
1 :05  *Note:  made  mistake  with  tasking  matrix  so  Log  O  updates  it 
1 :05  Ask  Kent  about  bridge  capability  (Kent  acting  as  Engr) 

1 :05  *Took  picture  of  updated  tasking  matrix 
1 :05  Logi  returns  to  diagramming  in  butcher  paper 
1 :06  commander  obtains  map  from  IO 

Comd  asks  A  Ops  O  to  write  key  assigned  and  implied  tasks  in  prep  for  mission  analysis 
1 :08  briefing 

1 :08  Ops  O  writes  msn  statement  onboard  and  key  deductions 

Comd  asks  if  everyone  will  be  ready  in  5  mins.  States  that  the  person  still  working  can 
1 :08  call  the  meeting  when  ready 
1 :09  commander  calls  meeting  in  5  minutes 

1 :09  OPS  O  and  AOPS  O  start  putting  their  information  the  white  board 
1:11  FOO  leaves  room 

Comd  outlines  meeting  -  Mission  Analysis  Briefing 
-  Report  individual  findings 
1:13  -  As  a  group  discuss  skeleton  COA 

Beginning  of  meeting:  stated  by  Comd  want  to  synthesize  info  collected  over  last  hour 
1:13  -  Comd  listening  more  in  meeting  and  asking  questions.  States  interpretations  out  loud 
Comd  explicitly  states  assumptions:  e.g.  Enemy  is  fixed.  Also  written  on  white  board 
1:13  with  other  assumptions 

Each  person  spoke,  probed  by  Comd... discussion  led  to  modification  of  the  mission 
1:13  statement 

1:14  FOO  returns  to  room  with  coffee 
1:14  Msn  analysis  brief: 

1:14  IO  -  enemy  likelyu  COA 

1:14  A  Ops  O  -  implied  and  assigned  tasks 

1:14  Ops  O  -  key  deductions 

1:14  Then  develop  skeleton  COA  together 

1:14  Comd:  "Once  developed  skeleton  COA,  certain  level  of  commitment" 

1:14  Comd  notes  key  assumption  is  enemy  is  fixed 

1:14  Ops  O  -  msn,  implied  and  assigned  tasks  and  key  assumptions  (3)  and  msn  statement 

1:14  Comd  approves  msn  statement 

1:14  A  Ops  O  discusses  additional  factors  and  deductions 

*Comd  says  that  point  of  this  is  to  synchronize  info  so  all  have  same  info  going  into  COA 
1:14  development 

1:14  Comd  gives  skeleton  COA  including  assumptions  and  initial  thoughts  (Not  CPG!) 

1:14  Movement  and  effects  (destroy  HB  &  C2),  risks,  admin 
1:14  Says  will  come  back  with  criteria  for  success  in  a  minute 
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*Says  what  backup  COA  is  *  and  asks  10  to  look  into  it  (Alternate  COA!?  Is  alternate 
1:14  crossing  viable?) 

1:14  10  discusses  other  crossing  options  to  go  north 

group  meeting  starts--each  present  info 
10 

OPSO 

Logi 

AOPSO 

1:15  made  false  assumption  about  accessibility  of  highway  2 

Comd  sets  outline  for  rest  of  process  and  addresses  the  importance  of  common 
1 :32  understanding  (sit  awareness) 

Again  goes  through  everyone  to  see  if  they  have  anything  to  add  (meta-cog  check 
1 :35  maybe?) 

Comd  has  listened  to  everyone  and  begins  his  talk  -  gives  skeleton  COA  verbally  (this  is 
not  the  CPG,  Comd  is  just  giving  out  ideas) 

-  emphasizes  that  this  is  a  Skeleton  COA  and  therefore  open  to  discussion.  Repeats 
future  direction  (i.e.  mental  wargaming) 

-  outlines  risk 

-  Defines  which  COA  they  are  going  to  focus  on. 

-  Comd  will  come  back  with  CPG 

-  Comd  vocalizes  an  alternative  approach  just  in  case 

-  Conclusion:  Fairly  good  picture  so  team  can  proceed.  CPG  at  10:45 

-  In  the  meantime  think  about  advantages  and  risk  mitigation  if  take  different  bridge  ->  is 
1 :36  another  crossing  viable? 

1 :39  commander  begins  discussing  skeleton  COA 

Comd  says  come  back  @1045  for  CPG  (-2:00  ex  time) 

1:45  ends  1:45:00 

1 :45  Note:  mental  wargaming  already  happening  -  comd  asking  what  if?  So  what? 

Brief  done 

1 :47  SME  -  Start  wargaming:  what  if,  so  what? 

1 :48  meeting  ends;  commander  asks  group  to  evaluate  viability  of  skeleton  COA 
1 :54  break 

1 :57  Comd  in  another  room  preparing  CPG 

1 :57  OPS  O,  AOPS  O,  FOO,  10:  exploring  bridge  route  further  on  overlay  map 
2:00  Comd  off  doing  CPG 

2:00  Ops  O,  A  Ops  O,  Log  O  and  FOO  collaborating 
2:06  AOPSO,  OPSO,  FOO  and  Logi  at  map  discussing 
Comd  returns  to  give  CPG 

-  asks  for  follow  up:  recommendations,  conclusion:  doesn't  mitigate  risk 

-  CPG  given  verbally,  will  post  additional  info  later 

-  agrees  with  mission  statement,  assumptions  etc 

-  states  end-state,  criteria  for  success,  effects,  risk  acceptance,  CONOP 

-  Comd  used  term  'Situational  Awareness' 

2:11  -  Comd  gives  CONOP 

2:12  CPG  presentation  by  Comd  -  all  present 

2:12  Ops  O  recommending  not  taking  alternate  route 

2:12  CPG  -  mission,  applied  and  assigned  tasks  and  assumptions  confirmed 

2:12  Criteria  for  success  (5): 

2:12  Conduct  of  attack  rehearsals 

2:12  Use  of  FAST  CAM 

2:12  Combat  air  support 

2:12  Elimination  anti-armour 
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2:12  Max  comms  with  higher  and  recce  for  accurate  picture  of  enemy 
2:12  Effects  -  destroy  75%  of  HB 
2:12  Risk  acceptable 
2:12  CONOP 

2:12  Describe  by  phases  with  assets  (order  of  march) 

2:12  Decision  points 

2:13  commander  returns  to  room;  group  meeting  begins 

commander  gives  planning  guidance  and  criteria  for  success--5  criteria  for  success  are 
2:14  given 

2:21  commander  gives  CONOP 

Says  carry  forward  with  COA  development 
2:24  ends  2:24:00 

2:24  Says  let's  do  meta-check  then  go  off  and  do  COA  validation  @1300 

Absorb  this  as  a  team,  do  a  meta  check,  go  off  to  work  individually  (This  is  how  Comd 
2:24  concludes  CPG) 

2:25  Beginning  of  COA  development 
2:26  meeting  ends 

2:29  Logi  and  FOO  work  independently 

2:30  Ops  O  notes  that  comd  has  actually  given  them  more  than  skeleton  COA 

2:30  A  gap  in  the  skeleton  is  brought  to  the  Comd's  attention  therefore  sits  down  to  fix  it 

2:35  Creating  task  matrix  (Log  O) 

2:36  commander  suggests  to  Logi  that  he  starts  on  his  matrix 
2:36  Logi  leaves  room 

Comd  tells  everyone  to  think  about  wargaming.  Validation  brief  at  13:00,  drive  process 
2:39  forward 

2:40  Start  mental  wargaming  based  on  tasking  matrix 
2:40  Want  validation  brief  @1300 
2:40  Comd  wants  time  appreciation 

Cmdr  asks  for  info  to  be  presented  on  whiteboard 

-  AOPS  O:  time  depreciation 

-  OPS  O:  working  on  each  phase  (coordination  among  phases,  line  of  march, 
coordination  among  elements,  breakdown  into  subunits) 

-  LOG:  doing  the  matrix  on  butcher  paper  (using  the  CONOP  and  his  personal  notes  that 
2:42  he  made  during  the  CPG 

2:45  Logi  working  on  matrix 

2:50  FOO  is  taking  map  measurements  for  AOPSO 

Comd  telling  group  about  meeting  he  had  with  10  about  10  minutes  ago. 

2:53  -  Telling  them  new  info  and  thoughts  to  consider... leads  to  further  discussion 

Individuals  start  giving  Comd  what  they  have  done. 

2:54  Discussion 

2:57  FOO  asks  commander  question  to  clarify  an  issue 

Comd  brings  new  aspect  to  think  about  for  AOPS  O  in  terms  of  time  depreciation. 

2:58  Then  generally  announces  'this  is  the  self  check' 

2:59  Issue  with  terminology  ->  ask  SME 

2:59  Small  changes  are  being  made  (come  from  discussion) 

2:59  Have  been  doing  mental  wargaming  for  30  mins 
3:00  Mental  wargaming  for  about  last  20mins,  by  phase 
3:00  Logi  still  working  on  matrix 

3:01  commander,  OPSO  and  AOPSO  engaged  in  meta-cognitive  checks 
3:03  commander  clarifies  terminiology  with  Kent 
3:04  Foo  and  commander  clarify  process 
3:04  commander  clarifies  an  issue  with  Kent 
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Time  Observation 

Lunch  -  mental  wargaming  at  lunch?? 

3:09  What  its  ->  resulted  in  changes  in  tactical  plans 
3:09  Participants  finished  lunch  before  Observers 
3:10  Lunch 
3:10  Lunch  break 

4:00  Logi  and  AOPSO  working  on  matrix;  FOO  prepares  transparency  for  putting  over  map 
4:03  OPSO  asks  Kent  question 

4:04  Had  already  begun  the  process.  Another  butcher  board  -  drawing  a  map 
4:05  NASA  TLX 

4:09  group  completes  NASA-TLX  and  MOPS 
4:09  Logi  continues  work  on  matrix 
4:1 1  AOPS  O  working  with  10  on  matrix 

4:12  Comd,  FOO,  OPS  O:  discuss  attack  position  on  butcher  paper  map 
4:20  COA  validation  brief 

4:20  Comd  comments  on  great  job  on  mental  wargaming 
4:20  Ops  O  -  major  A  in  initial  plan 

4:20  A  Ops  O  presents 

4:20  Comd  going  to  present  CONOP  to  Comd  after  this  move  into  COA  development 
4:24  Validation  Briefing 
4:29  briefing  meeting  begins 

4:47  commander  changes  one  of  the  criteria  for  success 
4:49  meeting  ends 

4:54  commander,  10,  AOPSO  leave  room 
4:56  Presentation  of  orders 
4:56  Describe  op  in  phases  (4) 

4:56  Higher  Comd  gives  feedback 

4:56  Brief  High  Comd 
5:01  briefing  to  Kent  begins 
5:09  brifing  ends 

5:10  group  completes  NASA-TLX  and  MOPS 
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Annex  D:  SME  BARS 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  the  Mission  Analysis  Process 

Look  for: 

(1 )  The  commander  provided  the  necessary  guidance  to  orient  the  staff 

(2)  The  process  remained  synchronized  and  integrated  and  served  to  keep  the 
commander  and  staff  on  a  common  basis 

(3)  The  commander  and  staff  displayed  situational  awareness 

Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  commander’s  guidance 

Look  for: 


Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  Staff  Analysis  process 

Look  for: 

(1 )  The  staff  developed  appropriate  factors  and  key  deductions 

(2)  The  staff  displayed  individual  initiative  and  creativity  within  the  guidance 
provided  by  the  commander 

Observations: 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  COA  Evaluation/Validation  process 

Look  for: 

(1 )  The  staff  fully  evaluated  each  friendly  and  enemy  COA  collectively  and,  as 
required,  independently 

(2)  All  COAs  carried  forward  appeared  to  be  viable 


Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  working  documents  (e.g.,  overlays) 

Look  for: 


Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Overall  efficiency  team  when  applying  planning  process 

Look  for: 

(1 )  Commander  and  staff  efforts  were  complementary  throughout 

(2)  The  commander  and  staff  applied  all  necessary  steps  in  a  logical  sequence 

(3)  Commander  and  staff  continued  to  build  on  previous  work  as  the  plan  was 
developed 

Observations: 


Page  D-2 


Human systems®  Incorporated 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  selected  COA 

Look  for: 

(1)  Selected  COA  is  complete  and  viable  and  has  fully  accounted  for  the 
enemy’s  most  dangerous  COA 

(2)  Selected  COA  is  doctrinally  sound 


Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Quality  of  CO 

NOP 

Look  for: 

(1)  The  CONOP  fully  describes  the  planned  operation  from  beginning  to  end. 

(2)  The  CONOP  appears  to  synchronize  “effects”  and  the  commander  and 
staff  understand  the  implications  of  those  effects 


Observations: 


1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Overall  quality  of  Plan 

Look  for: 

(1)  The  plan  is  easily  understood  by  sub-unit  commanders 

(2)  The  plan  is  simple  yet  incorporates  flexibility 

(3)  The  plan  is  workable  and  adheres  to  the  principles  of  mission  command 

Observations: 
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1.  Poor 

2.  Marginal 

3.  Standard 

4.  Very 
Good 

5.Exceptional 

Overall  quality  of  Order 

Look  for: 

(1)  The  tools  used  (matrix  order,  groupings  and  task  matrix,  etc.)  reinforce 
critical  aspects  of  the  plan 

(2)  The  order  is  easily  understood  by  sub-unit  commanders 


Observations: 
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Annex  E:  Participant  Rating  Scales 


Please  rate  the  following  using  the  scales  provided: 

Ease  of  use  of  planning  process 


1.  Very  difficult  2.  Difficult  3.  Standard 

4.  Easy 

5.Very  easy 

Learnability  of  planning  process 

1 .  Very  difficult  to  2.  Difficult  to  learn  3.  Standard 

learn 

4.  Easy  to  learn 

5. Very  easy  to 
learn 

Efficiency  of  planning  process 

I.Poor  2.  Marginal  3.  Standard 

4.  Very  Good 

5. Exceptional 

Effectiveness  of  planning  process 

I.Poor  2.  Marginal  3.  Standard 

4.  Very  Good 

5. Exceptional 

Complexity  of  planning  process 

1 .  Very  complex  2.  Complex  3.  Standard 

4.  Simple 

5.Very  simple 

Suitability  of  planning  process  in  this  context 

1.  Very  unsuitable  2.  Unsuitable  3.  Standard 

4.  Suitable 

5.Very  suitable 

Overall  user  satisfaction  with  planning  process 

1 .  Very  unsatisfied  2.  Unsatisfied  3.  Standard 

4.  Satisfied 

5.Very  Satisfied 

User  trust/confidence  in  planning  process 

I.None  2.  Low  3.  Medium 

4.  High 

5.Very  high 

Reliability  of  planning  process  to  produce  a  good  plan 

1.  Very  unreliable  2.  Unreliable  3.  Standard 

4.  Reliable 

5.Very  reliable 

Potential  for  planning  process  to  lead  to  errors  in  overall  Plan 

1.  Very  high  2.  High  3.  Standard 

4.  Low 

5.Very  low 

Potential  criticality  of  errors  in  overall  Plan  that  may  be  associated  with  planning  process 

1.  Very  high  2.  High  3.  Standard 

4.  Low 

5.Very  low 

Suitability  of  planning  process  in  a  variety  of  operational  contexts 

1.  Very  unsuitable  2.  Unsuitable  3.  Standard 

4.  Suitable 

5.Very  suitable 

Uumansystems ®  Incorporated 
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User  trust/confidence  in  the  overall  Plan  to  complete  the  mission 


I.None  2.  Low  3.  Medium 

4.  High 

5.Very  high 

Reliability  the  overall  Plan  to  complete  the  mission 

1.  Very  unreliable  2.  Unreliable  3.  Standard 

4.  Reliable 

5.Very  reliable 

Accuracy  of  the  overall  Plan  in  completing  the  mission 

1.  Very  inaccurate  2.  Inaccurate  3.  Standard 

4.  Accurate 

5.Very  accurate 

Overall  quality  of  Plan 

1 .  Very  low  quality  2.  Low  quality  3.  Standard 

4.  High  quality 

5.Very  high  quality 

Realism  of 
scenario 

1 .  Very  unrealistic  2.  Unrealistic  3.  Standard 

4.  Realistic 

5.Very  realistic 

Realism  of  time  available  for  planning 

1 .  Very  unrealistic  2.  Unrealistic  3.  Standard 

4.  Realistic 

5.Very  realistic 

Degree  to  which  the  team  followed  the  process 

I.Notatall  2.  Partially  3.  Generally 

4.  Mostly 

5. Precisely 
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Annex  F:  Swimlane  OPP  &  IOPP  Diagrams 


Intuitive  Operations  Planning  Process 

(IOPP) 


Situational  Awareness 


Mission 

Receipt 


^"'^tflission  Analysis  ^ 


COA  Development 


T 


COA 

Validation 


Issue 

Orders 


Plan  “ 
Developm’t 


Time 
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Staff  Commander 


1 .0  Situational  Awareness 


k_ 

Q)  "O 


s_  CO 


Performed  continuously  by  all  personnel 
(Higher  Cmd;  Cdr;  Staff  and  Lower  Formations) 


Situation 
Model  of 
Battles  pace 
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3.0  COA  Development 


Specific  Situational  Awareness 
Aspects  to  focus  on: 

-  Assumptions  and  Critical  Points 
-  Meta-cognitive  Check 


Sufficient 
info  to 

initiate  COA 
develop¬ 
ment? 


Cdr  creates 

_ Ttme  _ ► 

skeleton  COA  / 

constrained?/.. 

Yes 

Criteria  for 

Yes 

Success 

Reinitiate 

Mission 

Analysis 


No 


Cdr  may  ask 

Staff  to  create 

skeleton  COA 

— 

/  Criteria  for 

Success 

Staff  Create 
Skeleton  COA 


No 


Approve 
Skeleton 
COA  / 
Criteria  for 
Success  ? 


Yes 


Staff  Refine  / 
Develop  COA 


Skeleton 
COA  given 
to  LF 
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4.0  OCA  Validation 


Specific  Stuational  Av\aneness  Aspects  to 
focus  on: 

-  Evaluate  Assirrptions  and  Critical  point 

-  Meta-cognitive  Check 

-  Any  changes  to  Q/vrVEnerry  Situation? 
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Overview  of  OPP 


INITIATION 

ORIENTATION 

COA  DEVELOPMENT 

DECISION 

PLAN  DEVELOPMENT 

PLAN  REVIEW 
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1 .0  Initiation 
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Staff  Commander 


2.0  Orientation 
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3.0  OCA  Development 


CD  -& 


Attend 

Information 

Brief 

A 


Review 

Commander's 

Planning 

Guidance 


>  Staff  Analysis 


Develop  Initial 
►  Enemy  and 
OatiOCAs 


^  Test  Viability  of  ^ 
QatiOCAs 


T 


Prepare  and 
Present 
Information 
Brief 


Provide  Further 
Guidance 


_j y 

Oomd/OOS 

Provides 

Further 

Guidance 
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Any  OCAs  that 
iK  need  to  be  — ^ 
refined? 


Yes 


No 


Continue  Staff 
Checks  and 
Analyses  of 
QatiOCAs 


Refine  OCAs 
Selected  by 
Commander 


Develop  New 
OCAs  Greeted 
by  Commander 


Compare  Qati 
OCAs 


V\forgarring 


Identify 

►  branches  and 
sequels 
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Staff  Commander 


4.0  Decision 


Attend  decision 

Comd  Selects 

brief 

W 

COA 

* 


Concept  of 
Operations 
Developed 
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5.0  Ran  Ce^elcprrent 
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Issue  Final  Han 
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Staff  Commander 


6.0  Plan  Review 


CD  -q 


To  Step  1.0 


No 
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Annex  G:  Sequence  of  OPP  &  IOPP  Steps  Performed 


IOPP  Yellow,  Day  1 

10 

PP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

2.1 

Mission  Receipt 

2.1 

Mission  Receipt 

1.1 

Mission  Receipt 

1.1 

Mission  Receipt 

Not  in 
IOPP 

Review  of  players' 
roles 

Not  in 
IOPP 

Review  IOPP 

1.4 

Gather  Planning  Tools 

1.2 

Issue  Initiating 

Directive 

1.3 

Awareness  of 

Battlespace 

Environment 

1.1 

Awareness  of 

Enemy  Situation 

1.2 

Issue/receive  initiating 
directive 

1.4 

Gather  Planning 

Tools 

1.1 

Awareness  of 
Enemy  Situation 

1.4 

Awareness  of  Other 
Military  and  Civilian 
Resources 

1.4 

Gather  Planning  Tools 

Not  in 
OPP 

Preparing  Initial 

Comd's  Guidance 

1.3 

Awareness  of 

Battlespace 

Environment 

1.3 

Awareness  of 

Battlespace 

Environment 

1.5 

Comd  issues  guidelines 

1.4 

Gather  Planning 

Tools 

2.2 

Review  factors 

2.4 

Impied  Tasks 

Not  in 
OPP 

Preparing  Initial  Comd's 
Guidance 

1.5 

Comd  issues 
Guidelines 

1.3 

Awareness  of 

Battlespace 

Environment 

1.3 

Awareness  of 

Battlespace 

Environment 

2.1 

Review  situation 

1.6 

Comd  makes  initial 
assessment  (timing) 

Not  in 
IOPP 

Review  of  players' 
roles 

2.4 

Impied  Tasks 

1.4 

Gather  Planning  Tools 

1.4 

Gather  Planning 

Tools 

Not  in 
IOPP 

Understanding 
met-cog  check 

2.6 

Staff  Write  Draft 
Mission  Statement 

2.2 

Review  Higher  Level 
Information 

2.1 

Staff  review  situation 

1.3 

Awareness  of 

Battlespace 

Environment 

1.3 

Awareness  of 

Battlespace 

Environment 

1.4 

Gather  Planning  Tools 

1.7 

Initial  Comd's 

Guidance 

1.2 

Awareness  of 

Own  Situation 

2.2 

Comd  reviewing 
factors 

1.7 

Initial  Comd's  Guidance 

1.6 

Comd  makes  initial 
assessment  (timing) 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

2.3 

Comd  gives 
planning  guidance 

1.3 

Awareness  of 

Battlespace 

Environment 

1.6 

Comd  makes  initial 
assessment  (timings) 

2.1 

Staff  review  situation 

Not  in 
IOPP 

Understanding 
met-cog  check 

Not  in 
IOPP 

As  Comd  is 
reviewing  factors 
and  notices 
something  that  may 
be  of  importance  to 
a  particular  member 
of  his  staff  he  draws 
their  attention  to  it 

1.7 

Initial  Comd's  Guidance 

2.2 

Review  higher  level 
info 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

2.4 

Impied  Tasks 

2.1 

Review  own  situation 

2.3 

Develop  own  info 
based  on  higher  level 
info 

1.3 

Awareness  of 

Battlespace 

Environment 

2.2 

Review  factors 

2.2 

Review  Higher  Level 
Information 

2.4 

Staff  develop  Mission 
Statement 

Not  in 
IOPP 

Understanding 
met-cog  check 

2.3 

Initial  Planning 
Guidance 

2.3 

Develop  own  information 
based  on  higher  level 
info 

3.3 

Develop  Enemy  COA 

2.2 

Review  Factors 

1.3 

Awareness  of 

Battlespace 

Environment 

2.4 

Develop  mission 
statement 

2.4 

Staff  develop  Mission 
Statement 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

1.1 

Awareness  of 

Enemy  Situation 

2.1 

Review  Situation 

2.3 

Develop  own  info 
based  on  higher  level 
info 

1 

Situational 

Awareness 

1.2 

Awareness  of  Own 
Situation 

2.5 

Prepare  mission 
analysis  brief 

3.2 

Staff  Analysis 

Not  in 
IOPP 

Review  of  players' 
roles 

1.4 

Awareness  of  Other 
Military  and  Civilian 
Resources 

2.1 

Review  Situation 

3.3 

Develop  Initial  Enemy 
and  Own  COAs 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

2.2 

Comd  reviewing 
factors 

2.2 

Review  Higher  Level 
Information 

2.5 

Prepare  Mission 
Analysis  Brief 

1.5 

Meta-cog  check 

1.1 

Awareness  of 

Enemy  Situation 

2.3 

Develop  own  information 
based  on  higher  level 
info 

2.6 

Mission  Analysis 
Briefing 

2.2 

Review  factors 

3.2.2. 1 

Staff  Working  on 
Skeleton  COA 

1.6 

Modify  initial 
assessment  (timings) 

2.7 

Additional  guidance 
by  Comd 

2.3 

Comd  gives 
planning  guidance 

1.3 

Awareness  of 

Battlespace 

Environment 

2.1 

Review  Situation 

2.9 

Finalize  mission 
statement 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

Not  in 
IOPP 

Comd  discussing 
with  Higher  Comd 
(Kent)  E  COA 

1.6 

Modify  initial 
assessment  (timings) 

2.8 

Comd  develops 

Comd's  planning 
guidance 

2.5 

Define  most  likely 
ECOA 

1.5 

Conduct  meta-cog 
check 

2.3 

Develop  own  information 
based  on  higher  level 
info 

2.10 

CPG 

2.3 

Comd  gives 
planning  guidance 

1.3 

Awareness  of 

Battlespace 

Environment 

2.6 

Mission  Analysis  Briefing 

2.9 

Finalize  mission 
statement 

1.1 

Awareness  of 

Own  Situation 

2.3 

Initial  Planning 
Guidance 

2.10 

Comd's  Planning 
Guidance 

2.10 

CPG 

1.5 

Conduct  meta-cog 
check 

1.1 

Awareness  of 

Enemy  Situation 

2.7 

Comd  gives  additional 
guidance 

3.1 

Staff  review  CPG 

2.3 

Comd  gives 
planning  guidance 

1.2 

Awareness  of 

Enemy  Situation 

3.3 

Develop  Initial  Enemy 
and  Own  COAs 

3.2 

Staff  Analysis 

2.2 

Review  factors 

2.2 

Review  factors 

3.2 

Staff  Analysis 

3.3 

Develop  Initial  Enemy 
and  Own  COAs 

2.3 

Comd  gives 
planning  guidance 

3.2.1 

Comd  thinking 
about  Skeleton 

COA 

3.3 

Develop  Initial  Enemy 
and  Own  COAs 

3.5 

Information  Brief 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

1 

Situational 

Awareness 

2.6 

Staff  working  on 
draft  mission 
statement 

3.5 

Information  Brief 

3.6 

Comd  provides 
further  guidance 

2.2 

Review 

Assumptions 

2.7 

Mission  Analysis 
Briefing 

3.6 

Comd  provides  further 
guidance 

3.8 

Comd  decides  if 

COAs  need  to  be 
refined 

2.3 

Comd  gives 
planning  guidance 

1.5 

Conduct  meta-cog 
check 

3.5 

Information  Brief 

3.9 

Refine  COAs 

2.4 

Develop  Info 
Outlined  in  IPG 
and  Implied  Tasks 

2.5 

Staff  define  most 
likely  E  COA  (during 
mission  analysis 
briefing) 

3.8 

COAs  need  to  be 
refined? 

3.11 

Develop  new  COAs 
as  directed  by  Comd 

1.3 

Awareness  of 

Battlespace 

Environment 

Not  in 
IOPP 

Staff  and  Comd 
develop  Skeleton 
COA 

3.7 

Staff  continue  checks 
and  analyses  of  own 

COA 

3.13 

Staff  Wargame 

2.4 

Develop  Info 
Outlined  in  IPG 
and  Implied  Tasks 

2.2 

Review 

Assumptions 

3.13 

Staff  Wargame 

3.9 

Refine  COAs 

1.3 

Awareness  of 

Battlespace 

Environment 

2.8 

Comd  approves 
Mission  Statement 

3.14 

Identify  branches  and 
sequels 

4.1 

Review 

validation/comparison 

information 

2.5 

Define  most  likely 
ECOA 

1.5 

Conduct  meta-cog 
check 

3.9 

Refine  COAs 

4.2 

Decision  Brief 

1.3 

Awareness  of 

Battlespace 

Environment 

3.2.1 

Comd  provides 
Skeleton  COA 

3.12 

Compare  own  COAs 

4.3 

Comd  selects  COA 

1.5 

Conduct  meta-cog 
check 

Not  in 
IOPP 

Comd  provides 
alternative  COA 

4.2 

Decision  Brief 

5.1 

Staff  further  refines 
Comd  intent  and 
concept 

2.2 

Review  Assigned 
Tasks 

1.5 

Conduct  meta-cog 
check 

4.3 

Comd  selects  COA 

5.2 

Comd  seeks  CONOP 
approval 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

2.4 

Develop  Info 
Outlined  in  IPG 
and  Implied  Tasks 

4.4 

Mental  Wargaming 

5.5 

Prepare  plan 

5.3 

Higher  Comd 
approves  of  CONOP 

1.2 

Awareness  of 

Own  Situation 

3.4 

Refine/develop 

COA 

5.13 

Issue  Final  Plan 

5.4 

Identify  and  resolve 
issues/shortfalls 

1.3 

Awareness  of 

Battlespace 

Environment 

Not  in 
IOPP 

Comd  preparing  for 
CPG 

5.5 

Prepare  Plan 

1.5 

Meta-cog  check 

2.4 

Develop  info 
outlined  in  IPG  and 
Implied  Tasks  with 
reference  to 
alternative  COA 

5.13 

Issue  Final  Plan 

1.3 

Awareness  of 

Battlespace 

Environment 

2.9 

Comd  provides 

Initial  Intent 

2.2 

Review  factors 

3.2.1 

Comd  gives  Criteria 
for  Success 

2.5 

Define  most  likely 
ECOA 

1 

Situational 

Awareness 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

4.9 

Comd  gives 

CONOP 

2.5 

Define  most  likely 
ECOA 

1.5 

Conduct  meta-cog 
check 

Not  in 
IOPP 

Review  IOPP 

3.4 

Refine/develop 

COA 

2.5 

Define  most  likely 
ECOA 

4.4 

Mental  Wargaming 

2.6 

Write  draft  mission 
statement 

4.2 

Staff  Estimates 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

1.5 

Conduct  meta-cog 
check 

Not  in 
IOPP 

Review  of  players' 
roles 

4.4 

Mental  Wargaming 

Not  in 
IOPP 

Discussion  of 
attack  (remember 
this  is  a  defensive 
scenario) 

4.7 

Tweaking  of  COA 

Not  in 
IOPP 

Review  IOPP 

4.4 

Mental  Wargaming 

2.4 

Develop  Info 
Outlined  in  IPG 
and  Implied  Tasks 

1.5 

Conduct  meta-cog 
check 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

4.4 

Mental  Wargaming 

2.5 

Define  most  likely 
ECOA 

4.7 

Tweaking  of  COA 

1.4 

Awareness  of 
Military  and 

Civilian  Resources 

4.4 

Mental  Wargaming 

3.2.1 

Comd  gives 
Skeletoon  COA 

4.6 

Comd  approves 
validated  COA 

2.7 

Mission  Analysis 
Briefing 

4.10 

CONOP  to  higher 
Comd 

1 

Situational 

Awareness 

4.8 

Validation  briefing 

2.9 

Comd  provides 
Initial  Intent  (CPG) 

5.5 

Orders 

3.2.1 

Comd  gives 
Skeletoon  COA 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

OPP  Blue,  Day  1 

OPP  Yellow,  Day  2 

2.8 

Comd  approves  of 
Draft  Mission 
Statement 

3.2.1 

Comd  gives 
Skeletoon  COA 

3.4 

Refine/Develop 

COA 

2.5 

Define  Most  Likely 
ECOA 

3.4 

Refine/Develop 

COA 

1.4 

Awareness  of 

Other  Military  and 
Civilian  Resources 

3.4 

Refine/Develop 

COA 

1.3 

Awareness  of 

Battlespace 

Environment 

3.4 

Refine/Develop 

COA 

Not  in 
IOPP 

Cmdr  asks  for 
progress  check 

3.2.1 

Comd  gives 

Criteria  for 

Success 

3.5 

Finalize  COA 

1.5 

Conduct  meta-cog 
check 

4.2 

Staff  Estimates 

4.3 

Is  COA  viable? 

4.2 

Staff  Estimates 
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IOPP  Yellow,  Day  1 

IOPP  Blue,  Day  2 

4.4 

Mental 

Wargaming 

4.5 

Staff  and  Comd 
discuss  COA 
viability 

4.4 

Mental 

Wargaming 

4.6 

COA  Validated 

'4.10 

CONOP 

5.1 

Develop  Plan 

5.4 

Conitingency 

Plans 

5.1 

Develop  Plan 

5.5 

Orders 
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Annex  H:  Participants  Ratings  of  Planning  Products  and  Process 


name 

date 

leaseofuse 

2learnability 

3efficiency 

4effectivenes 

5complexity 

6suitability 

7usersatisf 

8usertrust 

9reliability 

Welsh 

19-Feb 

3 

2 

3 

4 

2 

1 

4 

4 

4 

Robertson 

19-Feb 

3 

3 

4 

4 

3 

4 

5 

5 

5 

Kellock 

19-Feb 

3 

3 

4 

4 

2 

4 

4 

5 

5 

Hall 

19-Feb 

3 

3 

4 

3 

2 

3 

2 

3 

3 

Stewart 

19-Feb 

3 

3 

2 

3 

2 

3 

3 

3 

4 

Alo 

Incomplete 

Bryden 

19-Feb 

4 

4 

4 

3 

4 

3 

4 

2 

3 

Van  Der 

Sluis 

19-Feb 

3 

3 

4 

4 

3 

5 

5 

5 

4 

Oneil 

19-Feb 

4 

4 

5 

4 

4 

5 

4 

4 

4 

Fitzgerald 

19-Feb 

4 

4 

4 

4 

3 

4 

4 

3 

4 

McMahon 

19-Feb 

4 

4 

4 

3 

4 

4 

4 

3 

3 

Johnston 

19-Feb 

2 

3 

4 

4 

2 

4 

4 

4 

4 
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name 

date 

lOerrorpotential 

1  lerrorcriticality 

12suitvarietyops 

13trustcompmsn 

14relycompmsn 

15accuracy 

16quality 

17realismscen 

18realismtime 

19degree 

Welsh 

19-Feb 

2 

3 

2 

4 

4 

4 

3 

2 

5 

4 

Robertson 

19-Feb 

4 

4 

4 

4 

5 

4 

4 

4 

4 

4 

Kellock 

19-Feb 

4 

3 

5 

4 

5 

5 

5 

5 

4 

5 

Hall 

19-Feb 

3 

3 

3 

3 

3 

3 

3 

4 

4 

4 

Stewart 

19-Feb 

4 

4 

3 

3 

4 

4 

4 

5 

4 

3 

Alo 

Incomplete 

Bryden 

19-Feb 

2 

3 

2 

3 

3 

3 

3 

5 

5 

4 

Van  Der 

Sluis 

19-Feb 

4 

3 

5 

5 

5 

4 

4 

5 

5 

4 

Oneil 

19-Feb 

4 

3 

4 

4 

4 

5 

4 

3 

3 

4 

Fitzgerald 

19-Feb 

3 

2 

3 

4 

4 

3 

3 

4 

4 

4 

McMahon 

19-Feb 

3 

3 

4 

3 

4 

4 

4 

4 

4 

4 

Johnston 

19-Feb 

2 

2 

4 

4 

4 

4 

3 

4 

3 

4 
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Annex  I:  Participant  Workload  Ratings 


name 

date 

scenario 

group 

role 

mental 

phys 

temporal 

perform 

effort 

frustration 

Welsh 

19-Feb 

Offense 

OPP 

Comd 

6.5 

5.5 

5.0 

2.5 

5.5 

5.0 

Robertson 

19-Feb 

Offense 

OPP 

Ops  0 

7.0 

7.0 

8.0 

3.0 

7.0 

3.0 

Kellock 

19-Feb 

Offense 

OPP 

Int  Officer 

3.0 

1.0 

4.0 

2.0 

5.0 

3.0 

Hall 

19-Feb 

Offense 

OPP 

Engineer 

6.5 

2.0 

7.0 

1.5 

6.0 

2.0 

Stewart 

19-Feb 

Offense 

OPP 

Fires 

6.0 

4.5 

6.5 

3.5 

6.0 

3.5 

Alo 

Incomplete 

Bryden 

Van  Der 

19-Feb 

B 

IOPP 

Comd 

8.5 

1.0 

2.0 

2.0 

5.0 

2.0 

Sluis 

19-Feb 

B 

Ex  King 

IOPP 

Ops  0 

5.5 

4.0 

4.5 

0.5 

6.0 

0.5 

Oneil 

19-Feb 

Cobra 

IOPP 

Int  Officer 

7.5 

3.5 

5.5 

7.0 

6.5 

5.5 

Fitzgerald 

19-Feb 

B 

IOPP 

Asst  Ops  0 

6.5 

2.0 

6.5 

2.5 

5.5 

2.5 

McMahon 

19-Feb 

IOPP 

Fires 

6.0 

1.5 

5.5 

2.0 

6.5 

3.0 

Johnston 

19-Feb 

B 

IOPP 

CIMIC/Ops 

5.0 

2.0 

5.0 

2.0 

3.0 

3.0 

name 

date 

eff-perf 

temp-frus 

temp-eff 

phys-frus 

perf-frus 

phys-temp 

phys-perf 

temp-ment 

Welsh 

19-Feb 

2 

1 

1 

2 

1 

2 

2 

2 

Robertson 

19-Feb 

2 

1 

1 

1 

1 

2 

2 

2 

Kellock 

19-Feb 

2 

2 

1 

2 

1 

2 

2 

2 

Hall 

19-Feb 

1 

1 

1 

1 

1 

2 

2 

1 

Stewart 

Alo 

19-Feb 

Incomplete 

1 

1 

1 

2 

1 

2 

2 

2 

Bryden 

Van  Der 

19-Feb 

2 

1 

1 

1 

1 

1 

2 

2 

Sluis 

19-Feb 

1 

1 

2 

2 

1 

2 

2 

2 

Oneil 

19-Feb 

2 

2 

1 

2 

2 

2 

2 

2 

Fitzgerald 

19-Feb 

2 

1 

1 

2 

1 

2 

2 

1 

McMahon 

19-Feb 

2 

1 

2 

2 

1 

2 

2 

2 

Johnston 

19-Feb 

2 

1 

2 

2 

1 

2 

2 

2 
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name 

date 

frus-eff 

perf-ment 

perf-temp 

ment-eff 

ment-phys 

eff-phys 

frus-ment 

Welsh 

19-Feb 

2 

1 

1 

2 

1 

1 

2 

Robertson 

19-Feb 

2 

2 

2 

1 

1 

2 

2 

Kellock 

19-Feb 

1 

1 

1 

1 

1 

1 

2 

Hall 

19-Feb 

2 

2 

2 

1 

1 

1 

2 

Stewart 

Alo 

19-Feb 

Incomplete 

2 

2 

2 

1 

1 

1 

1 

Bryden 

Van  Der 

19-Feb 

2 

2 

1 

1 

1 

1 

2 

Sluis 

19-Feb 

2 

2 

1 

2 

1 

1 

2 

Oneil 

19-Feb 

1 

2 

2 

1 

1 

1 

1 

Fitzgerald 

19-Feb 

1 

2 

2 

1 

1 

1 

2 

McMahon 

19-Feb 

2 

1 

1 

2 

1 

1 

2 

Johnston 

19-Feb 

2 

1 

1 

1 

1 

1 

2 
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